| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-24 | |||
| 09:11:27 | openstack | bugzilla.redhat.com bug 1090093 in libvirt "Improve error reporting when QEMU fails with incoming migration" [Unspecified,Closed: errata] - Assigned to jdenemar | |
| 09:13:12 | kashyap | bhagyashris: So I was told it's a distribution bug, so please file with Ubuntu. (Where the packager needs to pad the ROM with 0 bytes, using `dd`.) thanks, Dave Gilbert for the suggestion. | |
| 09:18:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add database schema upgrade check https://review.opendev.org/667047 | |
| 09:32:25 | openstackgerrit | Sam Morrison proposed openstack/nova stable/rocky: Fix double word hacking test https://review.opendev.org/665788 | |
| 09:33:30 | openstackgerrit | Sam Morrison proposed openstack/nova stable/queens: Fix double word hacking test https://review.opendev.org/665790 | |
| 09:39:45 | bhagyashris | kashyap: ok thank you! So for workaround I will set the kvm version same for both the machine and will try . | |
| 09:40:02 | kashyap | bhagyashris: Also see the guidance here: https://wiki.qemu.org/Features/Migration/Troubleshooting#Length_mismatch:_....:_...._in_.21.3D_..... | |
| 09:40:07 | kashyap | It does require work from distros. | |
| 09:43:02 | bhagyashris | kashyap: ok you mean it should install through bistro? | |
| 09:43:09 | bhagyashris | distro* | |
| 09:43:40 | kashyap | bhagyashris: I mean, there is an action item for Ubuntu QEMU packager to "pad the ROM" (it's a low-level thing) | |
| 09:50:48 | lwq | Hi there, I would like to discuss possible improvement of nova resize/migrate process - I mean the SCP part … Anyone with experience with this topic? :-) | |
| 10:20:19 | openstackgerrit | Merged openstack/python-novaclient master: Fix duplicate object description error https://review.opendev.org/666203 | |
| 10:22:31 | bhagyashris | kashyap: ohh ok thank you ! | |
| 10:35:22 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hacking: Resolve W503 (line break occurred before a binary operator) https://review.opendev.org/651555 | |
| 10:35:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hacking: Resolve E741 (ambiguous variable name) https://review.opendev.org/652103 | |
| 10:35:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: hacking: Resolve W605 (invalid escape sequence) https://review.opendev.org/652104 | |
| 10:35:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Use consistent URL regex substitution https://review.opendev.org/665949 | |
| 10:35:44 | openstackgerrit | Alex Xu proposed openstack/nova master: Assign the request_spec's network_metadata after create https://review.opendev.org/667061 | |
| 10:36:04 | openstackgerrit | Slawek Kaplonski proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/667035 | |
| 10:51:09 | alex_xu | stephenfin: something should be under your radar https://review.opendev.org/667061 | |
| 10:51:22 | stephenfin | alex_xu: Looking at it at the moment :) | |
| 11:45:31 | NewBruce3 | has anyone ever been able to live migrate VMs from CentOS (RDO) with SELinux to Ubuntu (OSA) with AppArmor? | |
| 11:50:04 | NewBruce | i am convinced there is a way to do it, reading into the libvirt docs, but nothing has worked so far | |
| 11:57:52 | kashyap | NewBruce: I don't think (read: "I'm pretty sure") you will find official libvirt docs for it :-) | |
| 11:58:28 | kashyap | If you're trying that, be aware that you might be left holding holding a bag of broken VM parts. | |
| 11:58:47 | kashyap | s/"holding holding"/holding/ | |
| 11:59:25 | kashyap | NewBruce: But ... if you _did_ succeed (or otherwise) might want to write a public blog post, with your exploits | |
| 12:10:06 | NewBruce | well you can ask mnaser how he did it ;) | |
| 12:11:13 | NewBruce | sorry, having a quick chat with some libvirt people who think im doing some horrible sorcery to get migration from centos to ubuntu in the first place and that is courtesy of mnaser but long story short seems… impossible? | |
| 12:13:15 | NewBruce | i expected to see a seclabel=selinux in the xml, but wasnt there but migrations fail…. | |
| 12:13:36 | NewBruce | Live Migration failure: unsupported configuration: Unable to find security driver for model selinux: libvirtError: unsupported configuration: Unable to find security driver for model selinux | |
| 12:19:55 | kashyap | NewBruce: Ah, I see you're on #virt, OFTC. (Haven't read the detail; buried in too many things) | |
| 12:20:53 | NewBruce | yeah, figured someone here might have tried before; but more likely the libvirt peoples would be up on it - seems this is a corner case; though surprising no one has ever moved from RDO to OSA before…. | |
| 12:21:00 | kashyap | Your machine types on source and destination won't match -- one of your main blockers... | |
| 12:21:32 | NewBruce | i beleive that is what mnaser has helped patch for us (wasnt around when he did that) | |
| 12:22:22 | NewBruce | but live migrating from RDO -> OSA definitely works for us; ive done it thousands of times now on another site - its just this setup im with now has SELinux on the RDO side which is a pain on the Ubuntu side as there are horrible bugs on Ubuntu with SELinux | |
| 12:22:54 | NewBruce | like not being able to create new users (which we can work around…. but still) | |
| 12:23:09 | NewBruce | running SELinux on Ubuntu is not a preferred long-term solution | |
| 12:24:04 | kashyap | Right, as I just noted on #virt; you do need some way of recompiling QEMU on Ubuntu, based on CentOS QEMU source (for the machine types) | |
| 12:24:16 | NewBruce | which it appears we have | |
| 12:24:32 | NewBruce | … should we move this over to #virt? :D | |
| 12:33:35 | kashyap | Yeah. | |
| 12:42:21 | mnaser | kashyap, NewBruce: pretty much deconstructed the srpm from redhat and added all the patches to the Deb which included all the machine typed etc to make it possible | |
| 12:46:17 | kashyap | mnaser: Wauw; have to admit, that's one really burdensome source you'll have to maintain | |
| 12:46:28 | NewBruce | according to the folks ok #virsh mnaser this makes you certifiable! :D | |
| 12:47:28 | mnaser | kashyap: I did tell the NewBruce and co that this will survive one single live migration and we reinstall the original binaries right after and it will block live migrations after | |
| 12:48:00 | mnaser | So the goal was just to get the one live migration done, reinstall original qemu and then on any full reboot they get the Ubuntu machine type | |
| 12:49:16 | kashyap | mnaser: If I may ask, is the goal to get off of CentOS altogether? | |
| 13:14:47 | efried_pto | sean-k-mooney: Is https://blueprints.launchpad.net/nova/+spec/libvirt-video-device-models code complete since https://review.opendev.org/#/c/647733/ has merged? | |
| 13:15:37 | efried_pto | sean-k-mooney: I see there's some minor cleanup, but that doesn't look critical. And something on the glance side? | |
| 13:25:13 | openstackgerrit | Merged openstack/os-traits master: add libvirt image metadata traits https://review.opendev.org/652996 | |
| 13:25:16 | openstackgerrit | Merged openstack/os-traits master: add GOP, NONE and VIRTIO grapics models https://review.opendev.org/665830 | |
| 13:33:44 | openstackgerrit | Merged openstack/nova master: Remove redundant group host setup https://review.opendev.org/665954 | |
| 13:33:53 | openstackgerrit | Merged openstack/nova master: [FUP] fix backleveling unit test for video models https://review.opendev.org/666919 | |
| 13:34:03 | openstackgerrit | Merged openstack/nova master: Mention [cinder]/cross_az_attach in the AZ docs https://review.opendev.org/650456 | |
| 13:41:01 | mriedem | sean-k-mooney: so same host revert resize huh, who know neutron had a gate job for ovs hybrid plug... | |
| 13:41:12 | mriedem | we should probably throw that into nova's experimental queue to be run on-demand | |
| 13:41:31 | mriedem | s/should/could/ since that means someone would have to remember to run it | |
| 13:44:00 | sean-k-mooney | ya we should | |
| 13:45:45 | efried_pto | sean-k-mooney: see question above about libvirt-video-device-models? | |
| 13:46:50 | efried_pto | oh | |
| 13:47:18 | sean-k-mooney | sorry give me 5 mins | |
| 13:47:35 | efried | sure, no hurry. Meanwhile, I'll hammer you in -placement as well. | |
| 13:49:56 | mriedem | sean is also working on pushing a patch to try and fix the neutron gate, | |
| 13:50:04 | mriedem | unless we just go with this revert https://review.opendev.org/#/c/667035/2 | |
| 13:50:18 | mriedem | slaweq: looks like ^ makes the job pass now, i checked zuul on your neutron change that depends on the nova revrt | |
| 13:50:21 | mriedem | *Revert | |
| 13:51:26 | efried | that's way more important than my blueprint paperwork | |
| 13:51:41 | mriedem | slaweq: what i'd like to know is do we still get a network-vif-plugged event for ovs hybrid plug ports during same host revert resize at the time of plugging the vif (because apparently we don't when binding the port and the host doesn't change - which is different in behavior from how non-hybrid plug ovs ports work apparently) | |
| 13:51:58 | efried | (</ptl-hat> for pretty much any value of $that) | |
| 13:52:05 | sean-k-mooney | mriedem: actully we had a production outage of our internal cloud and i just spent the last 2 hours helping them fix it | |
| 13:52:32 | mriedem | sean-k-mooney: please oh please say it was because of this ovs hybrid plug patch :) | |
| 13:53:12 | sean-k-mooney | mriedem: so im about to start on a patch to fix the neutron issue but ill leave it up to you do we want to revert it and merge a fixed patch later or merge a fix and wait form me to write it | |
| 13:53:47 | sean-k-mooney | mriedem: no it was releated to messing up the neutorn policy.json that currpted the network info_cache and all vms nolonger had interfaces | |
| 13:54:14 | mriedem | sean-k-mooney: aha that sounds familiar (the same thing ovh fixed in stein) | |
| 13:54:24 | sean-k-mooney | yep | |
| 13:54:30 | sean-k-mooney | we have not backported it to 13 yet | |
| 13:54:30 | mriedem | apparently screwing up your neutron policy and blasting the nova info cache is something a lot of people do... | |
| 13:55:12 | sean-k-mooney | ya so i just figured out what db surgery is need and our devops folks are now doing it | |
| 13:55:33 | sean-k-mooney | so do we want to fast revert the event change | |
| 13:55:47 | mriedem | sean-k-mooney: for artom's patch i think we likely should land the revert and then work the same host thing into a combined patch | |
| 13:55:51 | mriedem | since you guys want to backport this | |
| 13:55:54 | sean-k-mooney | or will i write a patch quickly to skip waiting for event for same host resize | |
| 13:56:28 | mriedem | reverting gives time to sort out why same host resize events are different for this vif type than non-hybrid plug ovs - that inconsistency is really screwy | |
| 13:56:29 | sean-k-mooney | ok im fine with that i just didnt want to cause more revew burden for you/others | |
| 13:56:51 | mriedem | slaweq's revert patch's commit message likely needs work | |
| 13:57:10 | rouk | sean-k-mooney: i found the ceph issue and it fixed the nova issue, it was an MTU someone thought would be fun to drop to 1500 on a single switch of the 2 switches doing LACP to that host. | |
| 13:57:40 | rouk | so some ceph responses would just fail, cause 9k byte, and then nova would sit forever silently | |
| 13:58:56 | dansmith | rouk: ouch :) | |
| 13:59:31 | rouk | nova needs some logging of its check cycle in debug mode though, having to trace from rabbitmq heartbeats as the symptom is rough | |
| 13:59:50 | sean-k-mooney | rouk: im sure that was fun to debug | |
| 14:00:00 | sean-k-mooney | it explains the behavior you were seeing however | |
| 14:00:35 | rouk | 4 days of man hours tracing through from the rabbitmq server because im always told "its not the network". | |
| 14:01:00 | sean-k-mooney | im still slightly annoyed/upsent/concerned that the nova-compute agent basicaly hard locks when it cant connect to ceph to get the capsity fo the ceph rbd pool | |
| 14:01:09 | rouk | so, can i wishlist some debug messages for the check phases in nova's cycle? lol | |
| 14:01:54 | rouk | yeah, it would send the message to ceph, and the ceph client retries every 30 seconds or so, and random dice based on what primary connection the LAPC uses determines if it returns or not, and it hangs forever till it gets a response. | |
| 14:02:01 | sean-k-mooney | hehe if you can suggest what they should be we can proably add them | |
| 14:03:11 | rouk | issue was only found tcpdumping ceph, would see a 9k byte response out of ceph and the client wouldnt hear about it. | |
| 14:03:46 | rouk | well... im not aware of all the checks, would be good if it was even just a basic debug on what function it was checking, just to narrow down debug. | |