| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-04 | |||
| 21:23:55 | _jralbert | We recently upgraded from Train to Ussuri; since then, our compute nodes which rely on libvirt and vfio mdev to provide access to vGPU slices of physical GPUs can no longer start nova-compute after a reboot. The nova-compute code around recreating assigned mdevs appears to have changed fairly significantly between these releases, and I'm wondering if anyone here knows the details to chat about them before I get into creating a bug repor | |
| #openstack-nova - 2021-10-05 | |||
| 06:45:45 | gibi | _jralbert: you need bauzas to wake up (he is in EU timezone) | |
| 07:17:08 | bauzas | good morning Nova | |
| 07:17:38 | bauzas | gibi, _jralbert: what's up ? | |
| 07:17:58 | bauzas | oh the reboot case | |
| 07:18:22 | bauzas | there is an open bug I need to correctly care https://bugs.launchpad.net/nova/+bug/1900800 | |
| 08:37:37 | gibi | bauzas: I knew that you know more | |
| 08:38:33 | bauzas | gibi: I know you knew I know ;) | |
| 08:38:53 | gibi | :) | |
| 08:55:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM Trigger tests with new eventlet and dnspython https://review.opendev.org/c/openstack/nova/+/812456 | |
| 09:46:56 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Move live migration jobs back to voting https://review.opendev.org/c/openstack/nova/+/812392 | |
| 09:46:56 | opendevreview | Lee Yarwood proposed openstack/nova master: Revert "zuul: Skip block migration with attached volumes tests due to bug #1931702" https://review.opendev.org/c/openstack/nova/+/812473 | |
| 09:47:10 | lyarwood | ^ okay this is looking good now | |
| 09:47:20 | lyarwood | I can't believe I didn't see this earlier but there we go | |
| 09:47:50 | gibi | better later than never ;) | |
| 10:28:41 | bauzas | lyarwood: I promise I'll look at your changes | |
| 10:34:04 | bauzas | lyarwood: actually, I had a comment for the devstack change https://review.opendev.org/c/openstack/devstack/+/812391/4/lib/nova#302 | |
| 10:36:33 | lyarwood | done | |
| 11:24:49 | lyarwood | actually I'm going to drop for a while this afternoon, could use the air ahead of being locked down until surgery next Tuesday, back online this evening UK time | |
| 11:25:03 | lyarwood | if anything comes up with the initiator change feel free to respin before I'm back | |
| 13:12:51 | kashyap | gibi: For later: Does chengsheng's response on this help any bit? -- https://review.opendev.org/c/openstack/nova/+/762330/ | |
| 13:13:34 | kashyap | I'd like to make some time to get this to completion. The lack of these two newer and better APIs is causing subtle live migration bugs :-( | |
| 13:17:53 | gibi | kashyap: added to my queue but wont reach it today | |
| 13:18:02 | kashyap | Does not have to be today :) | |
| 13:19:05 | gibi | kashyap: but thank you for the ping, I already forgot about his patch | |
| 13:53:42 | bauzas | artom_: sean-k-mooney: looks to me some SRIOV live migration issue https://bugs.launchpad.net/nova/+bug/1944619 | |
| 13:54:56 | sean-k-mooney | ill take a look but this is technially hardwoar offloade ovs if they are usign switch dev | |
| 13:55:10 | artom_ | bauzas, hrmm, looks we're not cleaning up properly in some cases | |
| 13:55:11 | sean-k-mooney | currently we do not support live migration with vdpa | |
| 13:55:23 | artom_ | Though I don't like "provoke an error in pre_live_migration" | |
| 13:55:26 | bauzas | artom_: can we triage this one ? | |
| 13:55:32 | sean-k-mooney | so if they are usign the sriov nic agent they may not use switchdev mode | |
| 13:55:48 | sean-k-mooney | if they are using switchdev mode then they have to use ovs hardware offload instead | |
| 13:58:07 | artom | bauzas, it'd be cool if they could upload full nova compute logs that include the error, ideally without the failure being "provoked" manually | |
| 13:59:33 | bauzas | they gave us some pastebin https://paste.ubuntu.com/p/ThQmDYtdSS/ | |
| 13:59:36 | bauzas | artom: ^ | |
| 14:02:29 | artom | bauzas, yeah, I'm not a fan of them triggering the failure by manually creating the destination instance disk | |
| 14:02:47 | artom | But I suppose it *could* happen organically, so Nova needs to handle it correctly? | |
| 14:03:51 | bauzas | artom: here, I don't really want to discuss about the bug, but rather about its upstream triage | |
| 14:04:29 | bauzas | artom: if you think they need to give us some other verification, could you ask it in the bug report and punt it to the 'Incomplete' status ? | |
| 14:05:11 | artom | bauzas, valid, I suppose, then | |
| 14:05:31 | bauzas | artom: ack, could you then triage it ? | |
| 14:05:34 | artom | Tbh, I don't know the gritty hardware details to understand what sean-k-mooney was saying about various levels of support in variou backends | |
| 14:06:55 | sean-k-mooney | im currently responding to it to level set what shoudl be supproted and what is not ssupproted | |
| 14:06:57 | sean-k-mooney | Check the logs for: libvirt.libvirtError: Requested operation is not valid: PCI device 0000:03:04.1 is in use by driver QEMU, domain instance-00000001 | |
| 14:07:14 | sean-k-mooney | ^ that seams to be the error they are reporting that breaks the roolback | |
| 14:07:20 | sean-k-mooney | that is not related to sriov live migration | |
| 14:07:36 | sean-k-mooney | that is cause by previou incorrect unshleves and possible cold migrations | |
| 14:08:06 | sean-k-mooney | bauzas: artom is aredally working on a fix for the in use device issue | |
| 14:08:17 | bauzas | kk | |
| 14:08:36 | artom | The unshelve fix? It merged in master, currently sitting on Victoria in the backport train | |
| 14:09:06 | artom | Depending on lyarwood's "controversial" helper backport here: https://review.opendev.org/c/openstack/nova/+/791481/1 | |
| 14:09:13 | sean-k-mooney | artom: that is part of it yes | |
| 14:09:18 | sean-k-mooney | the unsleve fix | |
| 14:09:48 | sean-k-mooney | although gibi also has another upstream bug for it which i am looking for currently | |
| 14:14:29 | sean-k-mooney | oh actully no that was a different issue | |
| 14:14:53 | sean-k-mooney | so the in use port i am pretty sure was cause by the unshelve of a vm previously | |
| 14:15:04 | sean-k-mooney | its not related to the sriov live migration | |
| 14:29:23 | sean-k-mooney | bauzas: artom im 99% sure this is hardware offload ovs not standard sriov by the way | |
| 14:38:53 | artom | sean-k-mooney, what's the difference? | |
| 15:03:15 | sean-k-mooney | artom: they have very different code paths | |
| 15:03:49 | sean-k-mooney | artom: for standard sriov all the interface configuretion is doen by libvirt | |
| 15:03:51 | opendevreview | Merged openstack/nova master: nova-manage: Ensure mountpoint is passed when updating attachment https://review.opendev.org/c/openstack/nova/+/811713 | |
| 15:04:34 | sean-k-mooney | for hardware offload ovs os-vif need to create ovs ports and add representor netdevs to ovs and the neutron l2 agent or ovn need to configure ovs to do thinks like vlan tagging | |
| 15:05:04 | sean-k-mooney | so the neutron contol path gose via ovn or the l2 agent not the sriov nic agent | |
| 15:05:17 | sean-k-mooney | the nic is in swtichdev mode not in legacy | |
| 15:06:18 | sean-k-mooney | vlan enfromanet is done via ovs not libvirt and you can use geneve/vxlan networks not just flat/vlan | |
| 15:11:34 | sean-k-mooney | there are some other difference too but those are the main ones | |
| 15:12:59 | yoctozepto | frickler: as centos has 5.1 - did you check the qemu cmdline there? | |
| 15:13:05 | frickler | sean-k-mooney: continuing about the qemu on debian discussion: these are instances with the same flavor (256M ram) in ubuntu and debian https://paste.opendev.org/show/809797/ | |
| 15:13:26 | frickler | yoctozepto: didn't hold that job yet, can do tomorrow | |
| 15:13:34 | yoctozepto | frickler: ok | |
| 15:13:56 | yoctozepto | I feel it might be reserving an extra chunk of mem for no reason | |
| 15:14:04 | yoctozepto | as it specifies the size there twice explicitly | |
| 15:14:07 | frickler | sean-k-mooney: the machine types look similar, but the added memory-backend option looks suspicous. but I assume that is added by libvirt, not nova? | |
| 15:15:26 | yoctozepto | << If "-machine memory-backend" is explicitly provided, "-m X" value must match specified backend's size >> | |
| 15:15:44 | yoctozepto | so by those semantics, it's just an unwieldy cmdline parsing assumption | |
| 15:15:48 | frickler | (context: devstack jobs on debian are ooming when tempest runs non-serial, see https://review.opendev.org/c/openstack/devstack/+/789083 ) | |
| 15:15:51 | yoctozepto | and should not consume more | |
| 15:16:14 | yoctozepto | but who knows (-: | |
| 15:16:42 | bauzas | reminder, nova meeting in 44 mins here in this channel #openstack-nova | |
| 15:33:48 | sean-k-mooney | frickler: yoctozepto sorry was updating a bug reading back | |
| 15:35:07 | sean-k-mooney | frickler: the macine types are similar but different version pc-i440fx-4.2 vs pc-i440fx-5.2 | |
| 15:37:15 | sean-k-mooney | frickler: yoctozepto regarding sepcifying the memory twice | |
| 15:37:16 | sean-k-mooney | -machine pc-i440fx-5.2,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram -cpu qemu64 -m 256 -object memory-backend-ram,id=pc.ram,size=268435456 | |
| 15:37:55 | sean-k-mooney | the -m 256 is the old way to specify it and memory-backend=pc.ram -object memory-backend-ram,id=pc.ram,size=268435456 | |
| 15:38:07 | sean-k-mooney | is the new way but you should be able to specify both | |
| 15:38:19 | sean-k-mooney | without it increase the memory allcoation | |
| 15:39:42 | sean-k-mooney | yoctozepto: frickler those commandline should result in the same behavior as far as i can see but this delta is cause by differnt libvirt versions | |
| 15:39:49 | sean-k-mooney | the nova generated xml shoudl be identicaly | |
| 15:40:17 | sean-k-mooney | debian is using libvirt 7.0.0 and ubuntu is using 6.0.0 | |
| 15:46:50 | sean-k-mooney | frickler: do you have the actual job logs so we can compare the nova xmls and ensure they are the same | |
| 15:47:15 | sean-k-mooney | if the xml created by nova is the same in both cases this is a libvirt/qemu bug most likely | |
| 15:59:20 | bauzas | nova meeting starting in 1 min | |
| 16:00:20 | bauzas | #startmeeting nova | |
| 16:00:20 | opendevmeet | Meeting started Tue Oct 5 16:00:20 2021 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:20 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:20 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:30 | bauzas | howdy folks | |
| 16:00:39 | dansmith | o/ | |