| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-10 | |||
| 10:28:57 | kevko | debian version | |
| 10:28:59 | kevko | buster one | |
| 10:32:07 | sean-k-mooney | that meets our min libvirt if only barely https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L225 | |
| 10:32:40 | sean-k-mooney | can you provide teh xml that was generated so we can see where the type element was added | |
| 10:32:53 | sean-k-mooney | i assume this was an interface attach? | |
| 10:33:09 | kevko | sean-k-mooney: well, i'm running tempests against stack .. if i turn on debug .. will i see xml ? | |
| 10:33:16 | kevko | sean-k-mooney: yeah, interface attach | |
| 10:33:33 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support https://review.opendev.org/c/openstack/nova/+/808199 | |
| 10:33:41 | sean-k-mooney | yes if you trun on debug you see a about a 8 line segment of xml printed for attach | |
| 10:34:10 | sean-k-mooney | we print the full xml when startign the vm and just the bit we are appending for attach | |
| 10:34:55 | kevko | ok, give me 5 minutes cca | |
| 10:37:46 | kevko | sean-k-mooney: https://paste.opendev.org/show/809224/ | |
| 10:39:15 | kevko | sean-k-mooney: bigger log -> https://paste.opendev.org/show/809225/ | |
| 10:39:48 | sean-k-mooney | that is odd that looks normal to me | |
| 10:40:05 | sean-k-mooney | oh what is your qemu version | |
| 10:40:26 | sean-k-mooney | is it below 4.0.0 | |
| 10:40:46 | kevko | https://paste.opendev.org/show/809226/ | |
| 10:40:54 | kevko | nope, it's 5.2 | |
| 10:41:14 | sean-k-mooney | ok i was wondering if it was older then libvirt suported | |
| 10:41:40 | sean-k-mooney | thye xml we are generating looks corect to me | |
| 10:41:58 | sean-k-mooney | so i suspect the issue is in the qemu args that libvirt is creeating | |
| 10:42:12 | sean-k-mooney | have you looked in the qemu instance long | |
| 10:42:15 | sean-k-mooney | *log | |
| 10:42:26 | sean-k-mooney | in /var/log/libvirt/qemu/instace* | |
| 10:42:35 | sean-k-mooney | is there any lower level error there | |
| 10:42:56 | kevko | let me check | |
| 10:45:57 | kevko | sean-k-mooney: nothing https://paste.opendev.org/show/809227/ | |
| 10:46:44 | sean-k-mooney | odd i dont even see thte attach | |
| 10:46:57 | sean-k-mooney | although it might not log that if it does not work | |
| 10:49:00 | sean-k-mooney | kevko: other then comparing the full xml to the fragment to see if there is any delta (comparing to the exising interface) | |
| 10:49:14 | sean-k-mooney | im not sure we can be much more help this looks like a libvirt bug | |
| 10:49:40 | kevko | i'm trying to replace victoria libvrit container for wallaby | |
| 10:49:52 | kevko | let me check what will happen :P | |
| 10:50:44 | sean-k-mooney | it likely should work nova support a fairly wide range | |
| 10:50:53 | sean-k-mooney | and libvirt has a pretty stable api | |
| 10:55:14 | kevko | sean-k-mooney: success | |
| 10:55:18 | kevko | sean-k-mooney: it's working | |
| 10:55:24 | kevko | nice | |
| 10:55:36 | kevko | so minimal version 5.0.0 is buggy | |
| 10:55:36 | gibi | hm, this sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1835312 | |
| 10:55:45 | gibi | that points to https://bugzilla.redhat.com/show_bug.cgi?id=1832710 | |
| 10:55:54 | gibi | which has a lot of libvirt commits to fix | |
| 10:56:29 | gibi | so yeah, possible libvirt - qemu incompatibility | |
| 10:57:06 | kevko | aaaaa | |
| 10:57:27 | kevko | gibi: yeah, it looks like that is the same thing as i met | |
| 10:57:56 | sean-k-mooney | ya since libvirt dont actully maintian stable branchs upstream frocing the distros to do it your milage might vary on the min version | |
| 10:58:12 | sean-k-mooney | i dont think we actully test our min version in the ci | |
| 10:58:23 | sean-k-mooney | we choose it based on the api feature set | |
| 10:58:40 | opendevreview | Merged openstack/nova master: Describe the use of Review-Priority label https://review.opendev.org/c/openstack/nova/+/792357 | |
| 10:58:42 | sean-k-mooney | and remove any condtional check for api compatiablity below that level | |
| 10:59:26 | sean-k-mooney | most of our testing id done with whatever version is shiped with ubuntu 20.04 | |
| 11:00:19 | sean-k-mooney | which is 6.0.0 | |
| 11:01:23 | sean-k-mooney | kevko: so anything above 5.0.0 should be api compatibale based on teh feature we use but we have only tested with 6.0.0+ for victoria | |
| 11:05:14 | sean-k-mooney | huh https://bugzilla.redhat.com/show_bug.cgi?id=1832710 well that is fun its nice when we are not told about this type of fix... | |
| 11:14:23 | kevko | well, i'm using kolla debian images which are based on buster in victoria (wallaby is already bullseye ..) | |
| 11:15:20 | kevko | so 6+ should be ok | |
| 12:26:55 | kashyap | gibi: FOr that netdev_add thing - so long as you're not mixing "old libvirt and new QEMU" for a given distro, you should not hit that issue. | |
| 12:27:29 | kashyap | gibi: I.e. if you're using CentOS 8.2, just stick to libvirt and QEMU from 8.2; don't mix versions from 8.3 | |
| 12:42:01 | gibi | kashyap: it was kevko who had an env with the issue. I'm not sure about the host OS in kevko's env | |
| 12:42:42 | kashyap | gibi: Sorry, yes; noticed it after you responded. (Didn't read the KM-long scrollback, afraid/) | |
| 12:42:54 | gibi | no worries | |
| 12:43:04 | gibi | kevko solved the issue by using newer libvirt | |
| 13:50:49 | bauzas | melwitt: gibi: eventually +Wd https://review.opendev.org/c/openstack/placement/+/807014 for the create project bugfix, but with some nits | |
| 14:16:01 | gibi | bauzas: thanks will read shortly | |
| 14:27:58 | gibi | bauzas: I agree with your nits. thanks again | |
| 15:09:49 | bauzas | gibi: ack, no worries, it's just nits | |
| 15:10:27 | gibi | bauzas: have a nice weekend | |
| 15:11:10 | bauzas | you too | |
| 15:11:38 | bauzas | (but not the CI one ;) ) | |
| 15:17:47 | gibi | :) | |
| 15:19:08 | melwitt | bauzas: thanks, I will make a followup patch to fix the nits | |
| 15:20:15 | opendevreview | OpenStack Release Bot proposed openstack/os-vif stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808452 | |
| 15:20:19 | opendevreview | OpenStack Release Bot proposed openstack/os-vif stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808453 | |
| 15:20:24 | opendevreview | OpenStack Release Bot proposed openstack/os-vif master: Update master for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808454 | |
| 15:20:29 | opendevreview | OpenStack Release Bot proposed openstack/os-vif master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/os-vif/+/808455 | |
| 15:20:41 | gibi | melwitt: placment the patch just bounced from the gate with strange functional test failures... | |
| 15:21:01 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808456 | |
| 15:21:05 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808457 | |
| 15:21:20 | melwitt | gibi: 😩 oh no. I will look | |
| 15:21:26 | gibi | https://597ae34c2915b12741a5-9637172b828bef74a2ccde1fe8027212.ssl.cf1.rackcdn.com/807014/6/gate/openstack-tox-functional-py38/bc74f18/testr_results.html | |
| 15:21:34 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement master: Update master for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808458 | |
| 15:22:23 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808459 | |
| 15:22:31 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808460 | |
| 15:22:36 | gibi | I can reproduce it on master locally without your patch so this is probably due to requirement change ... :/ | |
| 15:22:39 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient master: Update master for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808461 | |
| 15:22:45 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/python-novaclient/+/808462 | |
| 15:23:28 | melwitt | gibi: oh hm | |
| 15:24:11 | gibi | it is not visible in https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly so it happened in the last ~5 days | |
| 15:26:11 | gibi | hm, the global req bumped oslo.db to 11.0.0 two days ago, that is a likely candidate | |
| 15:28:54 | melwitt | agree. aside, there likely isn't a cross-placement job on global bump. I wonder if we should add one | |
| 15:31:17 | gibi | OK I confirmed that those tests fail with oslo.db 11.0.0 but passing with oslo.db 10.0.0 (the previous version) | |
| 15:31:42 | gibi | yeah I don't think we have a placement functional job on the global req repo | |
| 15:31:43 | sean-k-mooney | i dont see one in https://github.com/openstack/requirements/blob/master/.zuul.d/cross-jobs.yaml | |
| 15:31:46 | gibi | but we should | |
| 15:32:22 | sean-k-mooney | we have a nova one but you are correct no placment one https://github.com/openstack/requirements/blob/master/.zuul.d/project.yaml#L42 | |
| 15:58:37 | gibi | ok I failed to figure out how to reformulate those tests with the new sqlalchemy and I have to drop soon. sorry | |
| 15:59:21 | melwitt | thanks gibi. I'm also trying to fix, will let you know if I find any success | |
| 15:59:30 | gibi | thanks | |
| 15:59:35 | gibi | and good luck | |
| 15:59:53 | melwitt | thanks. have a nice weekend o/ | |