| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-10 | |||
| 09:35:28 | bauzas | but then I'll provide a new change explaining what I'd like to have to helping contributors | |
| 09:35:40 | gibi | OK, I will check that | |
| 09:36:46 | bauzas | gibi: about the bug you asked me | |
| 09:37:06 | bauzas | it's honestly a bug but Wishlist looks good to me | |
| 09:37:40 | bauzas | as you said, given the instance was offloaded from the compute, it's no longer possible to ask nova to wait for deleting it | |
| 09:42:50 | gibi | I would be affraid to change the delete behavior of offloaded instances as the current behavior was there forever and it is not really leaking any resources (I checked ports and volumes are cleaned up) | |
| 09:43:01 | gibi | so probalby people out there are adapted to this | |
| 09:51:19 | bauzas | gibi: we can fix this by documenting :) | |
| 09:51:57 | gibi | I can accept a documentation patch for it sure | |
| 10:13:05 | kevko | type for 'vhost', expected: boolean | |
| 10:13:05 | kevko | Hi folks, in victoria I am hitting this -> /var/log/kolla/nova/nova-compute.log:2021-09-10 09:49:09.815 8 ERROR nova.virt.libvirt.driver [req-8082a663-ca30-4845-9e93-2429c500ba76 31031550a7c94928abbd9dfa8634fba1 7b2ac086026741aa82c6c9c96ded42ee - default default] [instance: cf3ae6f9-a971-4123-ae73-9b597364985a] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter | |
| 10:13:18 | kevko | is it bug ? | |
| 10:25:05 | sean-k-mooney | kevko: can you past the libvirt xml somewhere so we can see what was generated | |
| 10:25:19 | sean-k-mooney | also what libvirt verions are you useing | |
| 10:28:54 | kevko | sean-k-mooney: https://paste.opendev.org/show/809223/ | |
| 10:28:57 | kevko | debian version | |
| 10:28:59 | kevko | buster one | |
| 10:32:07 | sean-k-mooney | that meets our min libvirt if only barely https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L225 | |
| 10:32:40 | sean-k-mooney | can you provide teh xml that was generated so we can see where the type element was added | |
| 10:32:53 | sean-k-mooney | i assume this was an interface attach? | |
| 10:33:09 | kevko | sean-k-mooney: well, i'm running tempests against stack .. if i turn on debug .. will i see xml ? | |
| 10:33:16 | kevko | sean-k-mooney: yeah, interface attach | |
| 10:33:33 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support https://review.opendev.org/c/openstack/nova/+/808199 | |
| 10:33:41 | sean-k-mooney | yes if you trun on debug you see a about a 8 line segment of xml printed for attach | |
| 10:34:10 | sean-k-mooney | we print the full xml when startign the vm and just the bit we are appending for attach | |
| 10:34:55 | kevko | ok, give me 5 minutes cca | |
| 10:37:46 | kevko | sean-k-mooney: https://paste.opendev.org/show/809224/ | |
| 10:39:15 | kevko | sean-k-mooney: bigger log -> https://paste.opendev.org/show/809225/ | |
| 10:39:48 | sean-k-mooney | that is odd that looks normal to me | |
| 10:40:05 | sean-k-mooney | oh what is your qemu version | |
| 10:40:26 | sean-k-mooney | is it below 4.0.0 | |
| 10:40:46 | kevko | https://paste.opendev.org/show/809226/ | |
| 10:40:54 | kevko | nope, it's 5.2 | |
| 10:41:14 | sean-k-mooney | ok i was wondering if it was older then libvirt suported | |
| 10:41:40 | sean-k-mooney | thye xml we are generating looks corect to me | |
| 10:41:58 | sean-k-mooney | so i suspect the issue is in the qemu args that libvirt is creeating | |
| 10:42:12 | sean-k-mooney | have you looked in the qemu instance long | |
| 10:42:15 | sean-k-mooney | *log | |
| 10:42:26 | sean-k-mooney | in /var/log/libvirt/qemu/instace* | |
| 10:42:35 | sean-k-mooney | is there any lower level error there | |
| 10:42:56 | kevko | let me check | |
| 10:45:57 | kevko | sean-k-mooney: nothing https://paste.opendev.org/show/809227/ | |
| 10:46:44 | sean-k-mooney | odd i dont even see thte attach | |
| 10:46:57 | sean-k-mooney | although it might not log that if it does not work | |
| 10:49:00 | sean-k-mooney | kevko: other then comparing the full xml to the fragment to see if there is any delta (comparing to the exising interface) | |
| 10:49:14 | sean-k-mooney | im not sure we can be much more help this looks like a libvirt bug | |
| 10:49:40 | kevko | i'm trying to replace victoria libvrit container for wallaby | |
| 10:49:52 | kevko | let me check what will happen :P | |
| 10:50:44 | sean-k-mooney | it likely should work nova support a fairly wide range | |
| 10:50:53 | sean-k-mooney | and libvirt has a pretty stable api | |
| 10:55:14 | kevko | sean-k-mooney: success | |
| 10:55:18 | kevko | sean-k-mooney: it's working | |
| 10:55:24 | kevko | nice | |
| 10:55:36 | gibi | hm, this sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1835312 | |
| 10:55:36 | kevko | so minimal version 5.0.0 is buggy | |
| 10:55:45 | gibi | that points to https://bugzilla.redhat.com/show_bug.cgi?id=1832710 | |
| 10:55:54 | gibi | which has a lot of libvirt commits to fix | |
| 10:56:29 | gibi | so yeah, possible libvirt - qemu incompatibility | |
| 10:57:06 | kevko | aaaaa | |
| 10:57:27 | kevko | gibi: yeah, it looks like that is the same thing as i met | |
| 10:57:56 | sean-k-mooney | ya since libvirt dont actully maintian stable branchs upstream frocing the distros to do it your milage might vary on the min version | |
| 10:58:12 | sean-k-mooney | i dont think we actully test our min version in the ci | |
| 10:58:23 | sean-k-mooney | we choose it based on the api feature set | |
| 10:58:40 | opendevreview | Merged openstack/nova master: Describe the use of Review-Priority label https://review.opendev.org/c/openstack/nova/+/792357 | |
| 10:58:42 | sean-k-mooney | and remove any condtional check for api compatiablity below that level | |
| 10:59:26 | sean-k-mooney | most of our testing id done with whatever version is shiped with ubuntu 20.04 | |
| 11:00:19 | sean-k-mooney | which is 6.0.0 | |
| 11:01:23 | sean-k-mooney | kevko: so anything above 5.0.0 should be api compatibale based on teh feature we use but we have only tested with 6.0.0+ for victoria | |
| 11:05:14 | sean-k-mooney | huh https://bugzilla.redhat.com/show_bug.cgi?id=1832710 well that is fun its nice when we are not told about this type of fix... | |
| 11:14:23 | kevko | well, i'm using kolla debian images which are based on buster in victoria (wallaby is already bullseye ..) | |
| 11:15:20 | kevko | so 6+ should be ok | |
| 12:26:55 | kashyap | gibi: FOr that netdev_add thing - so long as you're not mixing "old libvirt and new QEMU" for a given distro, you should not hit that issue. | |
| 12:27:29 | kashyap | gibi: I.e. if you're using CentOS 8.2, just stick to libvirt and QEMU from 8.2; don't mix versions from 8.3 | |
| 12:42:01 | gibi | kashyap: it was kevko who had an env with the issue. I'm not sure about the host OS in kevko's env | |
| 12:42:42 | kashyap | gibi: Sorry, yes; noticed it after you responded. (Didn't read the KM-long scrollback, afraid/) | |
| 12:42:54 | gibi | no worries | |
| 12:43:04 | gibi | kevko solved the issue by using newer libvirt | |
| 13:50:49 | bauzas | melwitt: gibi: eventually +Wd https://review.opendev.org/c/openstack/placement/+/807014 for the create project bugfix, but with some nits | |
| 14:16:01 | gibi | bauzas: thanks will read shortly | |
| 14:27:58 | gibi | bauzas: I agree with your nits. thanks again | |
| 15:09:49 | bauzas | gibi: ack, no worries, it's just nits | |
| 15:10:27 | gibi | bauzas: have a nice weekend | |
| 15:11:10 | bauzas | you too | |
| 15:11:38 | bauzas | (but not the CI one ;) ) | |
| 15:17:47 | gibi | :) | |
| 15:19:08 | melwitt | bauzas: thanks, I will make a followup patch to fix the nits | |
| 15:20:15 | opendevreview | OpenStack Release Bot proposed openstack/os-vif stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808452 | |
| 15:20:19 | opendevreview | OpenStack Release Bot proposed openstack/os-vif stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808453 | |
| 15:20:24 | opendevreview | OpenStack Release Bot proposed openstack/os-vif master: Update master for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808454 | |
| 15:20:29 | opendevreview | OpenStack Release Bot proposed openstack/os-vif master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/os-vif/+/808455 | |
| 15:20:41 | gibi | melwitt: placment the patch just bounced from the gate with strange functional test failures... | |
| 15:21:01 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808456 | |
| 15:21:05 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808457 | |
| 15:21:20 | melwitt | gibi: 😩 oh no. I will look | |
| 15:21:26 | gibi | https://597ae34c2915b12741a5-9637172b828bef74a2ccde1fe8027212.ssl.cf1.rackcdn.com/807014/6/gate/openstack-tox-functional-py38/bc74f18/testr_results.html | |
| 15:21:34 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement master: Update master for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808458 | |
| 15:22:23 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808459 | |
| 15:22:31 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808460 | |
| 15:22:36 | gibi | I can reproduce it on master locally without your patch so this is probably due to requirement change ... :/ | |
| 15:22:39 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient master: Update master for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808461 | |