| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-16 | |||
| 15:18:40 | sean-k-mooney | doing it premtivly is better but i think we get back into a consitent state | |
| 15:18:55 | sean-k-mooney | ill test it before i apply your patch and let you know | |
| 15:20:03 | gibi | sean-k-mooney: cool | |
| 15:25:50 | sean-k-mooney | gibi: by the way if i confim that we did not free the pci device do we want to backport a fix to block detach the same way we backported a patch to block attach for older releases? | |
| 15:26:08 | sean-k-mooney | i.e. if the periodic does not fix it | |
| 15:29:42 | gibi | sean-k-mooney: I'm not sure. If people using it then blocking it would be bad | |
| 15:30:12 | sean-k-mooney | it cloud be done vai a conig option i guess like numa live migration | |
| 15:30:25 | sean-k-mooney | we can cross that bridge when we come to it | |
| 15:30:29 | gibi | yeah | |
| 15:45:31 | gibi | nova meeting starts in 15 minutes on #openstack-meeting-3 | |
| 16:30:55 | artom | Hey, if we set a bug as incomplete, can the reporter then switch it back once they come back with new logs? | |
| 16:31:35 | artom | My thinking is: to avoid other people looking at it needlessly, set it as incomplete, and let the reporter come back with the info. Is that how we do i? | |
| 16:31:38 | artom | it | |
| 16:31:40 | gibi | artom: they can switch it back | |
| 16:31:55 | gibi | just ask them to set it back to New | |
| 16:32:47 | openstackgerrit | Elod Illes proposed openstack/nova stable/rocky: Check cherry-pick hashes in pep8 tox target https://review.opendev.org/741492 | |
| 16:44:24 | artom | Is (anti-)affinity in placement a thing we track? | |
| 16:44:33 | sean-k-mooney | artom: no | |
| 16:45:03 | openstack | Launchpad bug 1886160 in OpenStack Compute (nova) "Anti-affinity instance creation failed and the scheduling node was incorrect." [Undecided,New] | |
| 16:45:03 | artom | Context: https://bugs.launchpad.net/nova/+bug/1886160 looks like ye olde "the late anti-affinity check on the host is super racy" problem | |
| 16:45:11 | artom | So I'm looking for something to tell them other than "tough luck" | |
| 16:45:43 | sean-k-mooney | how about dont use multi create if you also use numa or sriov | |
| 16:46:31 | sean-k-mooney | the fact we now precompute a series of alternate host insead of rescudling on retry actully makes anti affinity worse | |
| 16:53:57 | artom | Right, we're < 30 untriaged, and I need lunch | |
| 17:04:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add vTPM config support https://review.opendev.org/739209 | |
| 17:04:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Simplify '_create_domain' function https://review.opendev.org/739208 | |
| 17:04:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: crypto: Add support for creating, destroying vTPM secrets https://review.opendev.org/739211 | |
| 17:04:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: scheduler: Request vTPM trait based on flavor or image https://review.opendev.org/739210 | |
| 17:04:30 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Rename tests for '_create_guest_with_network' https://review.opendev.org/740464 | |
| 17:04:30 | openstackgerrit | Stephen Finucane proposed openstack/nova master: manager: Prevent compute startup on invalid vTPM config https://review.opendev.org/739212 | |
| 17:04:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Define constants in '_IntegratedTestBase' https://review.opendev.org/741281 | |
| 17:04:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Move single use constants to their callers https://review.opendev.org/741280 | |
| 17:04:32 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add 'PlacementHelperMixin', 'PlacementInstanceHelperMixin' https://review.opendev.org/741283 | |
| 17:04:32 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Remove 'test_servers.ServersTestBase' https://review.opendev.org/741282 | |
| 17:04:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add helpers for suspend, resume and reboot of server https://review.opendev.org/741285 | |
| 17:04:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Make '_IntegratedTestBase' subclass 'PlacementInstanceHelperMixin' https://review.opendev.org/741284 | |
| 17:04:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 17:04:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Pass context, instance to '_create_domain' https://review.opendev.org/741286 | |
| 17:04:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support https://review.opendev.org/739213 | |
| 17:04:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 17:04:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Reject non-spawn operations for vTPM https://review.opendev.org/741500 | |
| 17:38:55 | openstackgerrit | sean mooney proposed openstack/nova master: lookup nic feature by pci address https://review.opendev.org/739131 | |
| 18:44:17 | sean-k-mooney | gibi: some comments in the patch. im not sure the interface you are using for the new fucntions is correct | |
| 18:44:39 | sean-k-mooney | overall the logic seam pretty sound but its not quite how i would do it | |
| 19:03:07 | sean-k-mooney | gibi: ya so the perodic task does not free the claim and they are not freed by the detach without your patch | |
| 19:03:24 | sean-k-mooney | gibi: deleting the instance does | |
| 19:04:04 | sean-k-mooney | so right now if you detatch an sriov interface it leaks the pci device until the vm is deleted | |
| 19:04:30 | sean-k-mooney | im going to check if we fix it with live migrations or cold migrations | |
| 19:07:06 | sean-k-mooney | yes live migration frees the claims | |
| 19:11:12 | sean-k-mooney | cold migration does not fix the issue an result in claiming a vf on the dest host that is unused | |
| 19:11:48 | sean-k-mooney | shevle and unshelve also claims an extra vf | |
| 19:13:28 | sean-k-mooney | gibi: so it look like anything that uses move claims does not correct it but sriov live migration which intentionally does not use move claims and caluatles the required pci device by looking at the attached interfaces does correct the issue | |
| 19:14:43 | sean-k-mooney | ok now that i have tested my patch ill test your and see if it fixes it. | |
| 19:33:00 | openstackgerrit | Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration https://review.opendev.org/741529 | |
| 19:51:41 | sean-k-mooney | tobias-urdin: ^ that is not really a fix for the issue its just going to mask the race condition | |
| 19:55:38 | sean-k-mooney | i left a comment inline but you need to use the python bindings not virsh | |
| 20:51:30 | mnaser | hrm | |
| 20:51:56 | melwitt | mnaser: did you see my reply to you yesterday? | |
| 20:52:06 | mnaser | melwitt: oh yes, sorry, i mentally said "yes" | |
| 20:52:19 | melwitt | np, just wanted to make sure :) | |
| 20:52:33 | mnaser | my fun thing now is a vm that takes 725 seconds to provision | |
| 20:52:40 | melwitt | yikes | |
| 20:52:48 | mnaser | and the only strange thing is like | |
| 20:52:59 | mnaser | Creating image .... nothing from there until VM started | |
| 20:53:06 | mnaser | 12 minutes until it starts | |
| 20:53:34 | mnaser | and here's the thing.. the image it uses? it's 1 megabyte. and it's cached locally (verified by sha1sum the image uuid) | |
| 20:53:47 | mnaser | https://www.irccloud.com/pastebin/s9lKj7n7/ | |
| 20:54:03 | mnaser | (that's grepping the nova-compute log for that specific vm) | |
| 20:54:11 | melwitt | hm, I've seen that hang before in cases where there was something like an environmental error where there was no OSD under the root | |
| 20:55:13 | melwitt | the 'rbd import' command can hang silently and I have not yet figured out a good way to time it out or anything | |
| 20:55:44 | melwitt | but in your case it's not stuck, it just takes forever | |
| 20:56:19 | mnaser | melwitt: well and also, sorry i should clarify, its not even using ceph | |
| 20:56:24 | mnaser | so local storage, so should be trivial? | |
| 20:56:38 | melwitt | should be... | |
| 20:56:59 | mnaser | and afaik the port plugging should happen after the vm get started (then resumed when the callback arrives) | |
| 20:57:10 | mnaser | which means it isn't waiting on neutron | |
| 20:58:26 | mnaser | but this is a case where like a single compute node is starting up 15 vms with 5 ports each potentially | |
| 21:05:57 | melwitt | mnaser: looks like the "Instance spawned successfully." is after port plugging and everything, it's after it's 100% done | |
| 21:06:40 | mnaser | melwitt: i thought the pause/resume on start that happens was for networking to get plumbed? | |
| 21:06:50 | melwitt | if you're able to turn on DEBUG, you could look for this log marker, this marks the end of the creating image piece https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L4070-L4073 | |
| 21:07:07 | openstackgerrit | Merged openstack/nova master: libvirt: Simplify '_create_domain' function https://review.opendev.org/739208 | |
| 21:07:17 | mnaser | i think i might have to switch to debug | |
| 21:07:40 | melwitt | yeah I'm still looking re: what the pause/resume happens. I'm not familiar with that other than knowing it happens | |
| 21:08:22 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Reintroduce native QEMU iSCSI volume support https://review.opendev.org/741546 | |
| 21:08:22 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Convert libvirt_volume_drivers to a dict https://review.opendev.org/741545 | |
| 21:08:26 | melwitt | yeah looks like resume is after plugging https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L6554 | |
| 21:08:40 | mnaser | so technically its not waiting for neutron.. | |
| 21:09:02 | melwitt | it is, right? that message is the resume, after plugging no? | |
| 21:09:45 | melwitt | (Based on seeing this before the resume https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L6513) | |
| 21:09:49 | mnaser | melwitt: 2020-07-16 20:37:40.144 => vm started, 2020-07-16 20:37:40.283 => vm paused, 2020-07-16 20:37:40.360 => vm resumed | |
| 21:10:27 | mnaser | anyhow we also run default vif_plugging_timeout which is 300s so it would have bombed then (there are logs of other vms bombing) | |
| 21:11:02 | melwitt | oh, hm | |
| 21:11:41 | mnaser | so i wonder the fact other threads are waiting for things are causing it to stall out | |
| 21:13:00 | melwitt | yeah... I guess we have seen things like that, where we had to dial down the concurrent builds setting | |
| 21:13:42 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/732717 | |
| 21:14:38 | melwitt | there was also a bug around some libvirt driver green thread proxying but it was fixed awhile back https://review.opendev.org/677736 | |
| 21:18:47 | mnaser | melwitt: interesting, that's not in this stein deployemnt | |
| 21:19:13 | melwitt | yeah, the backports all got nacked awhile back :\ | |
| 21:19:26 | mnaser | yep, i see that | |
| 21:19:33 | mnaser | seems like train came out with that | |