| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-10 | |||
| 02:41:15 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 06:08:09 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 07:00:49 | admin1 | seeking info on what happens during a migration .. say i have h1 ( hypervisor1 ) and h2 ( hypervisor2 ) .. when I want to migrate vm1 on h1 -> h2, what makes the ssh conection .. is it h1 nova -> h2 nova .. or h1 libvirt -> h2 libvirt or h1 root -> h2 root .. | |
| 07:24:31 | gibi | bauzas: o/ friendly reminder that we need to land melwitt's placemnt fix in Xena RC1 https://review.opendev.org/c/openstack/placement/+/807014 | |
| 07:25:08 | gibi | melwitt, dansmith: we are out of EU cores for now, so could you please look at a small doc patch before RC1 https://review.opendev.org/c/openstack/nova/+/807564 ? | |
| 07:25:38 | gibi | melwitt, dansmith: also the reno prelude is ready https://review.opendev.org/c/openstack/nova/+/807786 | |
| 07:56:14 | bauzas | gibi: yeah I know and I started to review it | |
| 07:56:26 | bauzas | but y'know how it's a bit difficult | |
| 07:56:42 | bauzas | + loving escalations during the last 2 days :( | |
| 07:57:17 | gibi | bauzas: thanks. feel free to ask questions about the patch I should be able to answer them | |
| 07:57:21 | gibi | ahh escalations. :/ | |
| 07:59:11 | bauzas | don't know what happended | |
| 07:59:31 | bauzas | maybe because of the moon or because of puppets | |
| 07:59:43 | bauzas | but looks like we have a lot of customers asking us to help them | |
| 08:00:01 | gibi | maybe they are back from summer vacation :) | |
| 08:01:16 | bauzas | they're not French :p | |
| 08:08:41 | gibi | then they are just nice and waited for the French to return from their summer vacation :P | |
| 08:28:07 | admin1 | one more question .. if cold migration works but live does not -- ceph backend .. where do i need to start looking .. the nova logs ( debug true ) does not provide any actionable insight .. | |
| 09:30:42 | gibi | bauzas: do you agree with my triage here https://bugs.launchpad.net/nova/+bug/1939856 that it is probably a new feature not a bugfix? | |
| 09:34:44 | bauzas | gibi: fwiw, I +wd https://review.opendev.org/c/openstack/nova/+/792357 | |
| 09:35:04 | gibi | bauzas: thanks | |
| 09:35:08 | bauzas | so we can start to use the new label | |
| 09:35:28 | bauzas | but then I'll provide a new change explaining what I'd like to have to helping contributors | |
| 09:35:40 | gibi | OK, I will check that | |
| 09:36:46 | bauzas | gibi: about the bug you asked me | |
| 09:37:06 | bauzas | it's honestly a bug but Wishlist looks good to me | |
| 09:37:40 | bauzas | as you said, given the instance was offloaded from the compute, it's no longer possible to ask nova to wait for deleting it | |
| 09:42:50 | gibi | I would be affraid to change the delete behavior of offloaded instances as the current behavior was there forever and it is not really leaking any resources (I checked ports and volumes are cleaned up) | |
| 09:43:01 | gibi | so probalby people out there are adapted to this | |
| 09:51:19 | bauzas | gibi: we can fix this by documenting :) | |
| 09:51:57 | gibi | I can accept a documentation patch for it sure | |
| 10:13:05 | kevko | Hi folks, in victoria I am hitting this -> /var/log/kolla/nova/nova-compute.log:2021-09-10 09:49:09.815 8 ERROR nova.virt.libvirt.driver [req-8082a663-ca30-4845-9e93-2429c500ba76 31031550a7c94928abbd9dfa8634fba1 7b2ac086026741aa82c6c9c96ded42ee - default default] [instance: cf3ae6f9-a971-4123-ae73-9b597364985a] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter | |
| 10:13:05 | kevko | type for 'vhost', expected: boolean | |
| 10:13:18 | kevko | is it bug ? | |
| 10:25:05 | sean-k-mooney | kevko: can you past the libvirt xml somewhere so we can see what was generated | |
| 10:25:19 | sean-k-mooney | also what libvirt verions are you useing | |
| 10:28:54 | kevko | sean-k-mooney: https://paste.opendev.org/show/809223/ | |
| 10:28:57 | kevko | debian version | |
| 10:28:59 | kevko | buster one | |
| 10:32:07 | sean-k-mooney | that meets our min libvirt if only barely https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L225 | |
| 10:32:40 | sean-k-mooney | can you provide teh xml that was generated so we can see where the type element was added | |
| 10:32:53 | sean-k-mooney | i assume this was an interface attach? | |
| 10:33:09 | kevko | sean-k-mooney: well, i'm running tempests against stack .. if i turn on debug .. will i see xml ? | |
| 10:33:16 | kevko | sean-k-mooney: yeah, interface attach | |
| 10:33:33 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support https://review.opendev.org/c/openstack/nova/+/808199 | |
| 10:33:41 | sean-k-mooney | yes if you trun on debug you see a about a 8 line segment of xml printed for attach | |
| 10:34:10 | sean-k-mooney | we print the full xml when startign the vm and just the bit we are appending for attach | |
| 10:34:55 | kevko | ok, give me 5 minutes cca | |
| 10:37:46 | kevko | sean-k-mooney: https://paste.opendev.org/show/809224/ | |
| 10:39:15 | kevko | sean-k-mooney: bigger log -> https://paste.opendev.org/show/809225/ | |
| 10:39:48 | sean-k-mooney | that is odd that looks normal to me | |
| 10:40:05 | sean-k-mooney | oh what is your qemu version | |
| 10:40:26 | sean-k-mooney | is it below 4.0.0 | |
| 10:40:46 | kevko | https://paste.opendev.org/show/809226/ | |
| 10:40:54 | kevko | nope, it's 5.2 | |
| 10:41:14 | sean-k-mooney | ok i was wondering if it was older then libvirt suported | |
| 10:41:40 | sean-k-mooney | thye xml we are generating looks corect to me | |
| 10:41:58 | sean-k-mooney | so i suspect the issue is in the qemu args that libvirt is creeating | |
| 10:42:12 | sean-k-mooney | have you looked in the qemu instance long | |
| 10:42:15 | sean-k-mooney | *log | |
| 10:42:26 | sean-k-mooney | in /var/log/libvirt/qemu/instace* | |
| 10:42:35 | sean-k-mooney | is there any lower level error there | |
| 10:42:56 | kevko | let me check | |
| 10:45:57 | kevko | sean-k-mooney: nothing https://paste.opendev.org/show/809227/ | |
| 10:46:44 | sean-k-mooney | odd i dont even see thte attach | |
| 10:46:57 | sean-k-mooney | although it might not log that if it does not work | |
| 10:49:00 | sean-k-mooney | kevko: other then comparing the full xml to the fragment to see if there is any delta (comparing to the exising interface) | |
| 10:49:14 | sean-k-mooney | im not sure we can be much more help this looks like a libvirt bug | |
| 10:49:40 | kevko | i'm trying to replace victoria libvrit container for wallaby | |
| 10:49:52 | kevko | let me check what will happen :P | |
| 10:50:44 | sean-k-mooney | it likely should work nova support a fairly wide range | |
| 10:50:53 | sean-k-mooney | and libvirt has a pretty stable api | |
| 10:55:14 | kevko | sean-k-mooney: success | |
| 10:55:18 | kevko | sean-k-mooney: it's working | |
| 10:55:24 | kevko | nice | |
| 10:55:36 | kevko | so minimal version 5.0.0 is buggy | |
| 10:55:36 | gibi | hm, this sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1835312 | |
| 10:55:45 | gibi | that points to https://bugzilla.redhat.com/show_bug.cgi?id=1832710 | |
| 10:55:54 | gibi | which has a lot of libvirt commits to fix | |
| 10:56:29 | gibi | so yeah, possible libvirt - qemu incompatibility | |
| 10:57:06 | kevko | aaaaa | |
| 10:57:27 | kevko | gibi: yeah, it looks like that is the same thing as i met | |
| 10:57:56 | sean-k-mooney | ya since libvirt dont actully maintian stable branchs upstream frocing the distros to do it your milage might vary on the min version | |
| 10:58:12 | sean-k-mooney | i dont think we actully test our min version in the ci | |
| 10:58:23 | sean-k-mooney | we choose it based on the api feature set | |
| 10:58:40 | opendevreview | Merged openstack/nova master: Describe the use of Review-Priority label https://review.opendev.org/c/openstack/nova/+/792357 | |
| 10:58:42 | sean-k-mooney | and remove any condtional check for api compatiablity below that level | |
| 10:59:26 | sean-k-mooney | most of our testing id done with whatever version is shiped with ubuntu 20.04 | |
| 11:00:19 | sean-k-mooney | which is 6.0.0 | |
| 11:01:23 | sean-k-mooney | kevko: so anything above 5.0.0 should be api compatibale based on teh feature we use but we have only tested with 6.0.0+ for victoria | |
| 11:05:14 | sean-k-mooney | huh https://bugzilla.redhat.com/show_bug.cgi?id=1832710 well that is fun its nice when we are not told about this type of fix... | |
| 11:14:23 | kevko | well, i'm using kolla debian images which are based on buster in victoria (wallaby is already bullseye ..) | |
| 11:15:20 | kevko | so 6+ should be ok | |
| 12:26:55 | kashyap | gibi: FOr that netdev_add thing - so long as you're not mixing "old libvirt and new QEMU" for a given distro, you should not hit that issue. | |
| 12:27:29 | kashyap | gibi: I.e. if you're using CentOS 8.2, just stick to libvirt and QEMU from 8.2; don't mix versions from 8.3 | |
| 12:42:01 | gibi | kashyap: it was kevko who had an env with the issue. I'm not sure about the host OS in kevko's env | |
| 12:42:42 | kashyap | gibi: Sorry, yes; noticed it after you responded. (Didn't read the KM-long scrollback, afraid/) | |
| 12:42:54 | gibi | no worries | |
| 12:43:04 | gibi | kevko solved the issue by using newer libvirt | |
| 13:50:49 | bauzas | melwitt: gibi: eventually +Wd https://review.opendev.org/c/openstack/placement/+/807014 for the create project bugfix, but with some nits | |