| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-10 | |||
| 10:41:58 | sean-k-mooney | so i suspect the issue is in the qemu args that libvirt is creeating | |
| 10:42:12 | sean-k-mooney | have you looked in the qemu instance long | |
| 10:42:15 | sean-k-mooney | *log | |
| 10:42:26 | sean-k-mooney | in /var/log/libvirt/qemu/instace* | |
| 10:42:35 | sean-k-mooney | is there any lower level error there | |
| 10:42:56 | kevko | let me check | |
| 10:45:57 | kevko | sean-k-mooney: nothing https://paste.opendev.org/show/809227/ | |
| 10:46:44 | sean-k-mooney | odd i dont even see thte attach | |
| 10:46:57 | sean-k-mooney | although it might not log that if it does not work | |
| 10:49:00 | sean-k-mooney | kevko: other then comparing the full xml to the fragment to see if there is any delta (comparing to the exising interface) | |
| 10:49:14 | sean-k-mooney | im not sure we can be much more help this looks like a libvirt bug | |
| 10:49:40 | kevko | i'm trying to replace victoria libvrit container for wallaby | |
| 10:49:52 | kevko | let me check what will happen :P | |
| 10:50:44 | sean-k-mooney | it likely should work nova support a fairly wide range | |
| 10:50:53 | sean-k-mooney | and libvirt has a pretty stable api | |
| 10:55:14 | kevko | sean-k-mooney: success | |
| 10:55:18 | kevko | sean-k-mooney: it's working | |
| 10:55:24 | kevko | nice | |
| 10:55:36 | gibi | hm, this sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1835312 | |
| 10:55:36 | kevko | so minimal version 5.0.0 is buggy | |
| 10:55:45 | gibi | that points to https://bugzilla.redhat.com/show_bug.cgi?id=1832710 | |
| 10:55:54 | gibi | which has a lot of libvirt commits to fix | |
| 10:56:29 | gibi | so yeah, possible libvirt - qemu incompatibility | |
| 10:57:06 | kevko | aaaaa | |
| 10:57:27 | kevko | gibi: yeah, it looks like that is the same thing as i met | |
| 10:57:56 | sean-k-mooney | ya since libvirt dont actully maintian stable branchs upstream frocing the distros to do it your milage might vary on the min version | |
| 10:58:12 | sean-k-mooney | i dont think we actully test our min version in the ci | |
| 10:58:23 | sean-k-mooney | we choose it based on the api feature set | |
| 10:58:40 | opendevreview | Merged openstack/nova master: Describe the use of Review-Priority label https://review.opendev.org/c/openstack/nova/+/792357 | |
| 10:58:42 | sean-k-mooney | and remove any condtional check for api compatiablity below that level | |
| 10:59:26 | sean-k-mooney | most of our testing id done with whatever version is shiped with ubuntu 20.04 | |
| 11:00:19 | sean-k-mooney | which is 6.0.0 | |
| 11:01:23 | sean-k-mooney | kevko: so anything above 5.0.0 should be api compatibale based on teh feature we use but we have only tested with 6.0.0+ for victoria | |
| 11:05:14 | sean-k-mooney | huh https://bugzilla.redhat.com/show_bug.cgi?id=1832710 well that is fun its nice when we are not told about this type of fix... | |
| 11:14:23 | kevko | well, i'm using kolla debian images which are based on buster in victoria (wallaby is already bullseye ..) | |
| 11:15:20 | kevko | so 6+ should be ok | |
| 12:26:55 | kashyap | gibi: FOr that netdev_add thing - so long as you're not mixing "old libvirt and new QEMU" for a given distro, you should not hit that issue. | |
| 12:27:29 | kashyap | gibi: I.e. if you're using CentOS 8.2, just stick to libvirt and QEMU from 8.2; don't mix versions from 8.3 | |
| 12:42:01 | gibi | kashyap: it was kevko who had an env with the issue. I'm not sure about the host OS in kevko's env | |
| 12:42:42 | kashyap | gibi: Sorry, yes; noticed it after you responded. (Didn't read the KM-long scrollback, afraid/) | |
| 12:42:54 | gibi | no worries | |
| 12:43:04 | gibi | kevko solved the issue by using newer libvirt | |
| 13:50:49 | bauzas | melwitt: gibi: eventually +Wd https://review.opendev.org/c/openstack/placement/+/807014 for the create project bugfix, but with some nits | |
| 14:16:01 | gibi | bauzas: thanks will read shortly | |
| 14:27:58 | gibi | bauzas: I agree with your nits. thanks again | |
| 15:09:49 | bauzas | gibi: ack, no worries, it's just nits | |
| 15:10:27 | gibi | bauzas: have a nice weekend | |
| 15:11:10 | bauzas | you too | |
| 15:11:38 | bauzas | (but not the CI one ;) ) | |
| 15:17:47 | gibi | :) | |
| 15:19:08 | melwitt | bauzas: thanks, I will make a followup patch to fix the nits | |
| 15:20:15 | opendevreview | OpenStack Release Bot proposed openstack/os-vif stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808452 | |
| 15:20:19 | opendevreview | OpenStack Release Bot proposed openstack/os-vif stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808453 | |
| 15:20:24 | opendevreview | OpenStack Release Bot proposed openstack/os-vif master: Update master for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808454 | |
| 15:20:29 | opendevreview | OpenStack Release Bot proposed openstack/os-vif master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/os-vif/+/808455 | |
| 15:20:41 | gibi | melwitt: placment the patch just bounced from the gate with strange functional test failures... | |
| 15:21:01 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808456 | |
| 15:21:05 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808457 | |
| 15:21:20 | melwitt | gibi: 😩 oh no. I will look | |
| 15:21:26 | gibi | https://597ae34c2915b12741a5-9637172b828bef74a2ccde1fe8027212.ssl.cf1.rackcdn.com/807014/6/gate/openstack-tox-functional-py38/bc74f18/testr_results.html | |
| 15:21:34 | opendevreview | OpenStack Release Bot proposed openstack/osc-placement master: Update master for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808458 | |
| 15:22:23 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808459 | |
| 15:22:31 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808460 | |
| 15:22:36 | gibi | I can reproduce it on master locally without your patch so this is probably due to requirement change ... :/ | |
| 15:22:39 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient master: Update master for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808461 | |
| 15:22:45 | opendevreview | OpenStack Release Bot proposed openstack/python-novaclient master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/python-novaclient/+/808462 | |
| 15:23:28 | melwitt | gibi: oh hm | |
| 15:24:11 | gibi | it is not visible in https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly so it happened in the last ~5 days | |
| 15:26:11 | gibi | hm, the global req bumped oslo.db to 11.0.0 two days ago, that is a likely candidate | |
| 15:28:54 | melwitt | agree. aside, there likely isn't a cross-placement job on global bump. I wonder if we should add one | |
| 15:31:17 | gibi | OK I confirmed that those tests fail with oslo.db 11.0.0 but passing with oslo.db 10.0.0 (the previous version) | |
| 15:31:42 | gibi | yeah I don't think we have a placement functional job on the global req repo | |
| 15:31:43 | sean-k-mooney | i dont see one in https://github.com/openstack/requirements/blob/master/.zuul.d/cross-jobs.yaml | |
| 15:31:46 | gibi | but we should | |
| 15:32:22 | sean-k-mooney | we have a nova one but you are correct no placment one https://github.com/openstack/requirements/blob/master/.zuul.d/project.yaml#L42 | |
| 15:58:37 | gibi | ok I failed to figure out how to reformulate those tests with the new sqlalchemy and I have to drop soon. sorry | |
| 15:59:21 | melwitt | thanks gibi. I'm also trying to fix, will let you know if I find any success | |
| 15:59:30 | gibi | thanks | |
| 15:59:35 | gibi | and good luck | |
| 15:59:53 | melwitt | thanks. have a nice weekend o/ | |
| 16:00:36 | gibi | same to you too eventually o/ | |
| 16:01:08 | melwitt | :) | |
| 16:03:12 | opendevreview | Artom Lifshitz proposed openstack/nova master: WIP: Gracefully power off guest on instance delete https://review.opendev.org/c/openstack/nova/+/808474 | |
| 16:29:04 | sean-k-mooney | artom: ok so that jsut add a call self._power_off_instance | |
| 16:29:15 | sean-k-mooney | are you sure we cant remove self._shutdown_instance(context, instance, bdms) | |
| 16:29:36 | artom | sean-k-mooney, maybe? Seems it does more than just power if off though... | |
| 16:31:53 | sean-k-mooney | it delete the volume attachments and detory the domain xml | |
| 16:31:58 | sean-k-mooney | that seam to be about it | |
| 16:32:09 | sean-k-mooney | presumable _cleanup_volumes would do the same | |
| 16:32:33 | sean-k-mooney | ok _cleanup_volumes does not handel the attachments | |
| 16:32:46 | sean-k-mooney | or calling terminat_connection | |
| 16:32:49 | sean-k-mooney | what about powerofof | |
| 16:32:52 | artom | lulz we just suck at reusing code, do we? | |
| 16:33:45 | sean-k-mooney | poweroff instance just calls driver.power off | |
| 16:34:11 | sean-k-mooney | what we proably shoudl do is remove the call to driver.destry in self._shutdown_instance | |
| 16:34:21 | dasm | hey team. i'm trying to wrap my head around placement for nova. When VCPUs number are changed? When VM is stopped? Or only when deleted? I found this old thread: http://lists.openstack.org/pipermail/openstack/2014-June/008022.html but not sure if it's still valid. | |
| 16:34:22 | sean-k-mooney | and have it call _power_off_isntace | |
| 16:35:42 | artom | sean-k-mooney, yeah, maybe. One thing I don't like about my current (admitedly very quickly hacked up) thing is that we're leaking libvirt driver stuff into the compute manager | |
| 16:35:56 | artom | Though I guess it *kinda* makes sense? | |
| 16:36:14 | opendevreview | melanie witt proposed openstack/placement master: Call Engine.execute() in func tests for oslo.db 11.0.0 https://review.opendev.org/c/openstack/placement/+/808477 | |