Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-16
16:31:38 artom it
16:31:40 gibi artom: they can switch it back
16:31:55 gibi just ask them to set it back to New
16:32:47 openstackgerrit Elod Illes proposed openstack/nova stable/rocky: Check cherry-pick hashes in pep8 tox target https://review.opendev.org/741492
16:44:24 artom Is (anti-)affinity in placement a thing we track?
16:44:33 sean-k-mooney artom: no
16:45:03 artom Context: https://bugs.launchpad.net/nova/+bug/1886160 looks like ye olde "the late anti-affinity check on the host is super racy" problem
16:45:03 openstack Launchpad bug 1886160 in OpenStack Compute (nova) "Anti-affinity instance creation failed and the scheduling node was incorrect." [Undecided,New]
16:45:11 artom So I'm looking for something to tell them other than "tough luck"
16:45:43 sean-k-mooney how about dont use multi create if you also use numa or sriov
16:46:31 sean-k-mooney the fact we now precompute a series of alternate host insead of rescudling on retry actully makes anti affinity worse
16:53:57 artom Right, we're < 30 untriaged, and I need lunch
17:04:28 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Simplify '_create_domain' function https://review.opendev.org/739208
17:04:28 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Add vTPM config support https://review.opendev.org/739209
17:04:29 openstackgerrit Stephen Finucane proposed openstack/nova master: scheduler: Request vTPM trait based on flavor or image https://review.opendev.org/739210
17:04:29 openstackgerrit Stephen Finucane proposed openstack/nova master: crypto: Add support for creating, destroying vTPM secrets https://review.opendev.org/739211
17:04:30 openstackgerrit Stephen Finucane proposed openstack/nova master: manager: Prevent compute startup on invalid vTPM config https://review.opendev.org/739212
17:04:30 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Rename tests for '_create_guest_with_network' https://review.opendev.org/740464
17:04:31 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Move single use constants to their callers https://review.opendev.org/741280
17:04:31 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Define constants in '_IntegratedTestBase' https://review.opendev.org/741281
17:04:32 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Remove 'test_servers.ServersTestBase' https://review.opendev.org/741282
17:04:32 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Add 'PlacementHelperMixin', 'PlacementInstanceHelperMixin' https://review.opendev.org/741283
17:04:33 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Make '_IntegratedTestBase' subclass 'PlacementInstanceHelperMixin' https://review.opendev.org/741284
17:04:33 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Add helpers for suspend, resume and reboot of server https://review.opendev.org/741285
17:04:34 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Pass context, instance to '_create_domain' https://review.opendev.org/741286
17:04:34 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Add emulated TPM support to Nova https://review.opendev.org/631363
17:04:35 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934
17:04:35 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support https://review.opendev.org/739213
17:04:40 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Reject non-spawn operations for vTPM https://review.opendev.org/741500
17:38:55 openstackgerrit sean mooney proposed openstack/nova master: lookup nic feature by pci address https://review.opendev.org/739131
18:44:17 sean-k-mooney gibi: some comments in the patch. im not sure the interface you are using for the new fucntions is correct
18:44:39 sean-k-mooney overall the logic seam pretty sound but its not quite how i would do it
19:03:07 sean-k-mooney gibi: ya so the perodic task does not free the claim and they are not freed by the detach without your patch
19:03:24 sean-k-mooney gibi: deleting the instance does
19:04:04 sean-k-mooney so right now if you detatch an sriov interface it leaks the pci device until the vm is deleted
19:04:30 sean-k-mooney im going to check if we fix it with live migrations or cold migrations
19:07:06 sean-k-mooney yes live migration frees the claims
19:11:12 sean-k-mooney cold migration does not fix the issue an result in claiming a vf on the dest host that is unused
19:11:48 sean-k-mooney shevle and unshelve also claims an extra vf
19:13:28 sean-k-mooney gibi: so it look like anything that uses move claims does not correct it but sriov live migration which intentionally does not use move claims and caluatles the required pci device by looking at the attached interfaces does correct the issue
19:14:43 sean-k-mooney ok now that i have tested my patch ill test your and see if it fixes it.
19:33:00 openstackgerrit Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration https://review.opendev.org/741529
19:51:41 sean-k-mooney tobias-urdin: ^ that is not really a fix for the issue its just going to mask the race condition
19:55:38 sean-k-mooney i left a comment inline but you need to use the python bindings not virsh
20:51:30 mnaser hrm
20:51:56 melwitt mnaser: did you see my reply to you yesterday?
20:52:06 mnaser melwitt: oh yes, sorry, i mentally said "yes"
20:52:19 melwitt np, just wanted to make sure :)
20:52:33 mnaser my fun thing now is a vm that takes 725 seconds to provision
20:52:40 melwitt yikes
20:52:48 mnaser and the only strange thing is like
20:52:59 mnaser Creating image .... nothing from there until VM started
20:53:06 mnaser 12 minutes until it starts
20:53:34 mnaser and here's the thing.. the image it uses? it's 1 megabyte. and it's cached locally (verified by sha1sum the image uuid)
20:53:47 mnaser https://www.irccloud.com/pastebin/s9lKj7n7/
20:54:03 mnaser (that's grepping the nova-compute log for that specific vm)
20:54:11 melwitt hm, I've seen that hang before in cases where there was something like an environmental error where there was no OSD under the root
20:55:13 melwitt the 'rbd import' command can hang silently and I have not yet figured out a good way to time it out or anything
20:55:44 melwitt but in your case it's not stuck, it just takes forever
20:56:19 mnaser melwitt: well and also, sorry i should clarify, its not even using ceph
20:56:24 mnaser so local storage, so should be trivial?
20:56:38 melwitt should be...
20:56:59 mnaser and afaik the port plugging should happen after the vm get started (then resumed when the callback arrives)
20:57:10 mnaser which means it isn't waiting on neutron
20:58:26 mnaser but this is a case where like a single compute node is starting up 15 vms with 5 ports each potentially
21:05:57 melwitt mnaser: looks like the "Instance spawned successfully." is after port plugging and everything, it's after it's 100% done
21:06:40 mnaser melwitt: i thought the pause/resume on start that happens was for networking to get plumbed?
21:06:50 melwitt if you're able to turn on DEBUG, you could look for this log marker, this marks the end of the creating image piece https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L4070-L4073
21:07:07 openstackgerrit Merged openstack/nova master: libvirt: Simplify '_create_domain' function https://review.opendev.org/739208
21:07:17 mnaser i think i might have to switch to debug
21:07:40 melwitt yeah I'm still looking re: what the pause/resume happens. I'm not familiar with that other than knowing it happens
21:08:22 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Convert libvirt_volume_drivers to a dict https://review.opendev.org/741545
21:08:22 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Reintroduce native QEMU iSCSI volume support https://review.opendev.org/741546
21:08:26 melwitt yeah looks like resume is after plugging https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L6554
21:08:40 mnaser so technically its not waiting for neutron..
21:09:02 melwitt it is, right? that message is the resume, after plugging no?
21:09:45 melwitt (Based on seeing this before the resume https://github.com/openstack/nova/blob/057891ac247fadfb9637d3e834f8b1ef3e39bc27/nova/virt/libvirt/driver.py#L6513)
21:09:49 mnaser melwitt: 2020-07-16 20:37:40.144 => vm started, 2020-07-16 20:37:40.283 => vm paused, 2020-07-16 20:37:40.360 => vm resumed
21:10:27 mnaser anyhow we also run default vif_plugging_timeout which is 300s so it would have bombed then (there are logs of other vms bombing)
21:11:02 melwitt oh, hm
21:11:41 mnaser so i wonder the fact other threads are waiting for things are causing it to stall out
21:13:00 melwitt yeah... I guess we have seen things like that, where we had to dial down the concurrent builds setting
21:13:42 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/732717
21:14:38 melwitt there was also a bug around some libvirt driver green thread proxying but it was fixed awhile back https://review.opendev.org/677736
21:18:47 mnaser melwitt: interesting, that's not in this stein deployemnt
21:19:13 melwitt yeah, the backports all got nacked awhile back :\
21:19:26 mnaser yep, i see that
21:19:33 mnaser seems like train came out with that
21:19:39 melwitt yes
21:19:51 openstackgerrit Lee Yarwood proposed openstack/nova master: Ensure source compute is up when confirming a resize https://review.opendev.org/699291
21:20:22 mnaser interestingly enough, all os_vif ops are taking almost 4-5s
21:20:51 mnaser and i think os_vif is just local only?
21:21:32 melwitt yeah, I think so
21:37:02 dansmith melwitt: ersum, thanks
21:37:40 melwitt yer welcome
21:42:22 openstackgerrit Stephen Finucane proposed openstack/nova master: api: Reject non-spawn operations for vTPM https://review.opendev.org/741500
21:42:23 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Add emulated TPM support to Nova https://review.opendev.org/631363
21:42:23 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Add docs for vTPM support https://review.opendev.org/739213
21:42:24 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934
21:56:32 openstackgerrit Tobias Urdin proposed openstack/nova master: [PoC] libvirt: Broadcast RARP post live migration https://review.opendev.org/741529

Earlier   Later