Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-10
07:56:26 bauzas but y'know how it's a bit difficult
07:56:42 bauzas + loving escalations during the last 2 days :(
07:57:17 gibi bauzas: thanks. feel free to ask questions about the patch I should be able to answer them
07:57:21 gibi ahh escalations. :/
07:59:11 bauzas don't know what happended
07:59:31 bauzas maybe because of the moon or because of puppets
07:59:43 bauzas but looks like we have a lot of customers asking us to help them
08:00:01 gibi maybe they are back from summer vacation :)
08:01:16 bauzas they're not French :p
08:08:41 gibi then they are just nice and waited for the French to return from their summer vacation :P
08:28:07 admin1 one more question .. if cold migration works but live does not -- ceph backend .. where do i need to start looking .. the nova logs ( debug true ) does not provide any actionable insight ..
09:30:42 gibi bauzas: do you agree with my triage here https://bugs.launchpad.net/nova/+bug/1939856 that it is probably a new feature not a bugfix?
09:34:44 bauzas gibi: fwiw, I +wd https://review.opendev.org/c/openstack/nova/+/792357
09:35:04 gibi bauzas: thanks
09:35:08 bauzas so we can start to use the new label
09:35:28 bauzas but then I'll provide a new change explaining what I'd like to have to helping contributors
09:35:40 gibi OK, I will check that
09:36:46 bauzas gibi: about the bug you asked me
09:37:06 bauzas it's honestly a bug but Wishlist looks good to me
09:37:40 bauzas as you said, given the instance was offloaded from the compute, it's no longer possible to ask nova to wait for deleting it
09:42:50 gibi I would be affraid to change the delete behavior of offloaded instances as the current behavior was there forever and it is not really leaking any resources (I checked ports and volumes are cleaned up)
09:43:01 gibi so probalby people out there are adapted to this
09:51:19 bauzas gibi: we can fix this by documenting :)
09:51:57 gibi I can accept a documentation patch for it sure
10:13:05 kevko type for 'vhost', expected: boolean
10:13:05 kevko Hi folks, in victoria I am hitting this -> /var/log/kolla/nova/nova-compute.log:2021-09-10 09:49:09.815 8 ERROR nova.virt.libvirt.driver [req-8082a663-ca30-4845-9e93-2429c500ba76 31031550a7c94928abbd9dfa8634fba1 7b2ac086026741aa82c6c9c96ded42ee - default default] [instance: cf3ae6f9-a971-4123-ae73-9b597364985a] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter
10:13:18 kevko is it bug ?
10:25:05 sean-k-mooney kevko: can you past the libvirt xml somewhere so we can see what was generated
10:25:19 sean-k-mooney also what libvirt verions are you useing
10:28:54 kevko sean-k-mooney: https://paste.opendev.org/show/809223/
10:28:57 kevko debian version
10:28:59 kevko buster one
10:32:07 sean-k-mooney that meets our min libvirt if only barely https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L225
10:32:40 sean-k-mooney can you provide teh xml that was generated so we can see where the type element was added
10:32:53 sean-k-mooney i assume this was an interface attach?
10:33:09 kevko sean-k-mooney: well, i'm running tempests against stack .. if i turn on debug .. will i see xml ?
10:33:16 kevko sean-k-mooney: yeah, interface attach
10:33:33 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support https://review.opendev.org/c/openstack/nova/+/808199
10:33:41 sean-k-mooney yes if you trun on debug you see a about a 8 line segment of xml printed for attach
10:34:10 sean-k-mooney we print the full xml when startign the vm and just the bit we are appending for attach
10:34:55 kevko ok, give me 5 minutes cca
10:37:46 kevko sean-k-mooney: https://paste.opendev.org/show/809224/
10:39:15 kevko sean-k-mooney: bigger log -> https://paste.opendev.org/show/809225/
10:39:48 sean-k-mooney that is odd that looks normal to me
10:40:05 sean-k-mooney oh what is your qemu version
10:40:26 sean-k-mooney is it below 4.0.0
10:40:46 kevko https://paste.opendev.org/show/809226/
10:40:54 kevko nope, it's 5.2
10:41:14 sean-k-mooney ok i was wondering if it was older then libvirt suported
10:41:40 sean-k-mooney thye xml we are generating looks corect to me
10:41:58 sean-k-mooney so i suspect the issue is in the qemu args that libvirt is creeating
10:42:12 sean-k-mooney have you looked in the qemu instance long
10:42:15 sean-k-mooney *log
10:42:26 sean-k-mooney in /var/log/libvirt/qemu/instace*
10:42:35 sean-k-mooney is there any lower level error there
10:42:56 kevko let me check
10:45:57 kevko sean-k-mooney: nothing https://paste.opendev.org/show/809227/
10:46:44 sean-k-mooney odd i dont even see thte attach
10:46:57 sean-k-mooney although it might not log that if it does not work
10:49:00 sean-k-mooney kevko: other then comparing the full xml to the fragment to see if there is any delta (comparing to the exising interface)
10:49:14 sean-k-mooney im not sure we can be much more help this looks like a libvirt bug
10:49:40 kevko i'm trying to replace victoria libvrit container for wallaby
10:49:52 kevko let me check what will happen :P
10:50:44 sean-k-mooney it likely should work nova support a fairly wide range
10:50:53 sean-k-mooney and libvirt has a pretty stable api
10:55:14 kevko sean-k-mooney: success
10:55:18 kevko sean-k-mooney: it's working
10:55:24 kevko nice
10:55:36 gibi hm, this sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1835312
10:55:36 kevko so minimal version 5.0.0 is buggy
10:55:45 gibi that points to https://bugzilla.redhat.com/show_bug.cgi?id=1832710
10:55:54 gibi which has a lot of libvirt commits to fix
10:56:29 gibi so yeah, possible libvirt - qemu incompatibility
10:57:06 kevko aaaaa
10:57:27 kevko gibi: yeah, it looks like that is the same thing as i met
10:57:56 sean-k-mooney ya since libvirt dont actully maintian stable branchs upstream frocing the distros to do it your milage might vary on the min version
10:58:12 sean-k-mooney i dont think we actully test our min version in the ci
10:58:23 sean-k-mooney we choose it based on the api feature set
10:58:40 opendevreview Merged openstack/nova master: Describe the use of Review-Priority label https://review.opendev.org/c/openstack/nova/+/792357
10:58:42 sean-k-mooney and remove any condtional check for api compatiablity below that level
10:59:26 sean-k-mooney most of our testing id done with whatever version is shiped with ubuntu 20.04
11:00:19 sean-k-mooney which is 6.0.0
11:01:23 sean-k-mooney kevko: so anything above 5.0.0 should be api compatibale based on teh feature we use but we have only tested with 6.0.0+ for victoria
11:05:14 sean-k-mooney huh https://bugzilla.redhat.com/show_bug.cgi?id=1832710 well that is fun its nice when we are not told about this type of fix...
11:14:23 kevko well, i'm using kolla debian images which are based on buster in victoria (wallaby is already bullseye ..)
11:15:20 kevko so 6+ should be ok
12:26:55 kashyap gibi: FOr that netdev_add thing - so long as you're not mixing "old libvirt and new QEMU" for a given distro, you should not hit that issue.
12:27:29 kashyap gibi: I.e. if you're using CentOS 8.2, just stick to libvirt and QEMU from 8.2; don't mix versions from 8.3
12:42:01 gibi kashyap: it was kevko who had an env with the issue. I'm not sure about the host OS in kevko's env
12:42:42 kashyap gibi: Sorry, yes; noticed it after you responded. (Didn't read the KM-long scrollback, afraid/)
12:42:54 gibi no worries
12:43:04 gibi kevko solved the issue by using newer libvirt
13:50:49 bauzas melwitt: gibi: eventually +Wd https://review.opendev.org/c/openstack/placement/+/807014 for the create project bugfix, but with some nits
14:16:01 gibi bauzas: thanks will read shortly
14:27:58 gibi bauzas: I agree with your nits. thanks again
15:09:49 bauzas gibi: ack, no worries, it's just nits
15:10:27 gibi bauzas: have a nice weekend
15:11:10 bauzas you too
15:11:38 bauzas (but not the CI one ;) )
15:17:47 gibi :)

Earlier   Later