| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-24 | |||
| 11:45:31 | NewBruce3 | has anyone ever been able to live migrate VMs from CentOS (RDO) with SELinux to Ubuntu (OSA) with AppArmor? | |
| 11:50:04 | NewBruce | i am convinced there is a way to do it, reading into the libvirt docs, but nothing has worked so far | |
| 11:57:52 | kashyap | NewBruce: I don't think (read: "I'm pretty sure") you will find official libvirt docs for it :-) | |
| 11:58:28 | kashyap | If you're trying that, be aware that you might be left holding holding a bag of broken VM parts. | |
| 11:58:47 | kashyap | s/"holding holding"/holding/ | |
| 11:59:25 | kashyap | NewBruce: But ... if you _did_ succeed (or otherwise) might want to write a public blog post, with your exploits | |
| 12:10:06 | NewBruce | well you can ask mnaser how he did it ;) | |
| 12:11:13 | NewBruce | sorry, having a quick chat with some libvirt people who think im doing some horrible sorcery to get migration from centos to ubuntu in the first place and that is courtesy of mnaser but long story short seems… impossible? | |
| 12:13:15 | NewBruce | i expected to see a seclabel=selinux in the xml, but wasnt there but migrations fail…. | |
| 12:13:36 | NewBruce | Live Migration failure: unsupported configuration: Unable to find security driver for model selinux: libvirtError: unsupported configuration: Unable to find security driver for model selinux | |
| 12:19:55 | kashyap | NewBruce: Ah, I see you're on #virt, OFTC. (Haven't read the detail; buried in too many things) | |
| 12:20:53 | NewBruce | yeah, figured someone here might have tried before; but more likely the libvirt peoples would be up on it - seems this is a corner case; though surprising no one has ever moved from RDO to OSA before…. | |
| 12:21:00 | kashyap | Your machine types on source and destination won't match -- one of your main blockers... | |
| 12:21:32 | NewBruce | i beleive that is what mnaser has helped patch for us (wasnt around when he did that) | |
| 12:22:22 | NewBruce | but live migrating from RDO -> OSA definitely works for us; ive done it thousands of times now on another site - its just this setup im with now has SELinux on the RDO side which is a pain on the Ubuntu side as there are horrible bugs on Ubuntu with SELinux | |
| 12:22:54 | NewBruce | like not being able to create new users (which we can work around…. but still) | |
| 12:23:09 | NewBruce | running SELinux on Ubuntu is not a preferred long-term solution | |
| 12:24:04 | kashyap | Right, as I just noted on #virt; you do need some way of recompiling QEMU on Ubuntu, based on CentOS QEMU source (for the machine types) | |
| 12:24:16 | NewBruce | which it appears we have | |
| 12:24:32 | NewBruce | … should we move this over to #virt? :D | |
| 12:33:35 | kashyap | Yeah. | |
| 12:42:21 | mnaser | kashyap, NewBruce: pretty much deconstructed the srpm from redhat and added all the patches to the Deb which included all the machine typed etc to make it possible | |
| 12:46:17 | kashyap | mnaser: Wauw; have to admit, that's one really burdensome source you'll have to maintain | |
| 12:46:28 | NewBruce | according to the folks ok #virsh mnaser this makes you certifiable! :D | |
| 12:47:28 | mnaser | kashyap: I did tell the NewBruce and co that this will survive one single live migration and we reinstall the original binaries right after and it will block live migrations after | |
| 12:48:00 | mnaser | So the goal was just to get the one live migration done, reinstall original qemu and then on any full reboot they get the Ubuntu machine type | |
| 12:49:16 | kashyap | mnaser: If I may ask, is the goal to get off of CentOS altogether? | |
| 13:14:47 | efried_pto | sean-k-mooney: Is https://blueprints.launchpad.net/nova/+spec/libvirt-video-device-models code complete since https://review.opendev.org/#/c/647733/ has merged? | |
| 13:15:37 | efried_pto | sean-k-mooney: I see there's some minor cleanup, but that doesn't look critical. And something on the glance side? | |
| 13:25:13 | openstackgerrit | Merged openstack/os-traits master: add libvirt image metadata traits https://review.opendev.org/652996 | |
| 13:25:16 | openstackgerrit | Merged openstack/os-traits master: add GOP, NONE and VIRTIO grapics models https://review.opendev.org/665830 | |
| 13:33:44 | openstackgerrit | Merged openstack/nova master: Remove redundant group host setup https://review.opendev.org/665954 | |
| 13:33:53 | openstackgerrit | Merged openstack/nova master: [FUP] fix backleveling unit test for video models https://review.opendev.org/666919 | |
| 13:34:03 | openstackgerrit | Merged openstack/nova master: Mention [cinder]/cross_az_attach in the AZ docs https://review.opendev.org/650456 | |
| 13:41:01 | mriedem | sean-k-mooney: so same host revert resize huh, who know neutron had a gate job for ovs hybrid plug... | |
| 13:41:12 | mriedem | we should probably throw that into nova's experimental queue to be run on-demand | |
| 13:41:31 | mriedem | s/should/could/ since that means someone would have to remember to run it | |
| 13:44:00 | sean-k-mooney | ya we should | |
| 13:45:45 | efried_pto | sean-k-mooney: see question above about libvirt-video-device-models? | |
| 13:46:50 | efried_pto | oh | |
| 13:47:18 | sean-k-mooney | sorry give me 5 mins | |
| 13:47:35 | efried | sure, no hurry. Meanwhile, I'll hammer you in -placement as well. | |
| 13:49:56 | mriedem | sean is also working on pushing a patch to try and fix the neutron gate, | |
| 13:50:04 | mriedem | unless we just go with this revert https://review.opendev.org/#/c/667035/2 | |
| 13:50:18 | mriedem | slaweq: looks like ^ makes the job pass now, i checked zuul on your neutron change that depends on the nova revrt | |
| 13:50:21 | mriedem | *Revert | |
| 13:51:26 | efried | that's way more important than my blueprint paperwork | |
| 13:51:41 | mriedem | slaweq: what i'd like to know is do we still get a network-vif-plugged event for ovs hybrid plug ports during same host revert resize at the time of plugging the vif (because apparently we don't when binding the port and the host doesn't change - which is different in behavior from how non-hybrid plug ovs ports work apparently) | |
| 13:51:58 | efried | (</ptl-hat> for pretty much any value of $that) | |
| 13:52:05 | sean-k-mooney | mriedem: actully we had a production outage of our internal cloud and i just spent the last 2 hours helping them fix it | |
| 13:52:32 | mriedem | sean-k-mooney: please oh please say it was because of this ovs hybrid plug patch :) | |
| 13:53:12 | sean-k-mooney | mriedem: so im about to start on a patch to fix the neutron issue but ill leave it up to you do we want to revert it and merge a fixed patch later or merge a fix and wait form me to write it | |
| 13:53:47 | sean-k-mooney | mriedem: no it was releated to messing up the neutorn policy.json that currpted the network info_cache and all vms nolonger had interfaces | |
| 13:54:14 | mriedem | sean-k-mooney: aha that sounds familiar (the same thing ovh fixed in stein) | |
| 13:54:24 | sean-k-mooney | yep | |
| 13:54:30 | mriedem | apparently screwing up your neutron policy and blasting the nova info cache is something a lot of people do... | |
| 13:54:30 | sean-k-mooney | we have not backported it to 13 yet | |
| 13:55:12 | sean-k-mooney | ya so i just figured out what db surgery is need and our devops folks are now doing it | |
| 13:55:33 | sean-k-mooney | so do we want to fast revert the event change | |
| 13:55:47 | mriedem | sean-k-mooney: for artom's patch i think we likely should land the revert and then work the same host thing into a combined patch | |
| 13:55:51 | mriedem | since you guys want to backport this | |
| 13:55:54 | sean-k-mooney | or will i write a patch quickly to skip waiting for event for same host resize | |
| 13:56:28 | mriedem | reverting gives time to sort out why same host resize events are different for this vif type than non-hybrid plug ovs - that inconsistency is really screwy | |
| 13:56:29 | sean-k-mooney | ok im fine with that i just didnt want to cause more revew burden for you/others | |
| 13:56:51 | mriedem | slaweq's revert patch's commit message likely needs work | |
| 13:57:10 | rouk | sean-k-mooney: i found the ceph issue and it fixed the nova issue, it was an MTU someone thought would be fun to drop to 1500 on a single switch of the 2 switches doing LACP to that host. | |
| 13:57:40 | rouk | so some ceph responses would just fail, cause 9k byte, and then nova would sit forever silently | |
| 13:58:56 | dansmith | rouk: ouch :) | |
| 13:59:31 | rouk | nova needs some logging of its check cycle in debug mode though, having to trace from rabbitmq heartbeats as the symptom is rough | |
| 13:59:50 | sean-k-mooney | rouk: im sure that was fun to debug | |
| 14:00:00 | sean-k-mooney | it explains the behavior you were seeing however | |
| 14:00:35 | rouk | 4 days of man hours tracing through from the rabbitmq server because im always told "its not the network". | |
| 14:01:00 | sean-k-mooney | im still slightly annoyed/upsent/concerned that the nova-compute agent basicaly hard locks when it cant connect to ceph to get the capsity fo the ceph rbd pool | |
| 14:01:09 | rouk | so, can i wishlist some debug messages for the check phases in nova's cycle? lol | |
| 14:01:54 | rouk | yeah, it would send the message to ceph, and the ceph client retries every 30 seconds or so, and random dice based on what primary connection the LAPC uses determines if it returns or not, and it hangs forever till it gets a response. | |
| 14:02:01 | sean-k-mooney | hehe if you can suggest what they should be we can proably add them | |
| 14:03:11 | rouk | issue was only found tcpdumping ceph, would see a 9k byte response out of ceph and the client wouldnt hear about it. | |
| 14:03:46 | rouk | well... im not aware of all the checks, would be good if it was even just a basic debug on what function it was checking, just to narrow down debug. | |
| 14:04:17 | rouk | right now the messages stop pretty high up, so the only thing to check is everything. | |
| 14:04:49 | sean-k-mooney | im guessing it was working just well enough for the tcp connection not to drop | |
| 14:05:10 | sean-k-mooney | if we got a disconnect that would proably have shown up in the logs | |
| 14:05:12 | rouk | any tcp/udp/icmp with RESPONSES under 1500 worked perfectly 100% of the time. | |
| 14:05:42 | sean-k-mooney | ya im not sure nova could figure that our however. | |
| 14:05:51 | sean-k-mooney | rouk: did you file a bug for this by the way? | |
| 14:06:02 | rouk | nope, since its my fault. | |
| 14:06:29 | rouk | if nova would give up on the ceph client after even 20 seconds it would be fine imo | |
| 14:06:30 | sean-k-mooney | well we proably should be making that call with a timeout and raising an error if we dont get the resopce | |
| 14:06:57 | rouk | packet loss ceph client hangs for 30 seconds per retry or so, but if the backend is not responding for even over 1 second, somethings up. | |
| 14:07:27 | rouk | yeah, anything over 1 second is bad, and 20 seconds imo is an ERROR tier. | |
| 14:07:46 | sean-k-mooney | ya i think as a general enchancement haveing a config option to control a timout on ceph calls would likely be a good thing | |
| 14:08:14 | rouk | should i fill it out somewhere in particular? | |
| 14:09:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove comments about mirroring changes to nova/cells/messaging.py https://review.opendev.org/667107 | |
| 14:09:09 | sean-k-mooney | rouk: https://bugs.launchpad.net/nova/+filebug | |
| 14:10:07 | sean-k-mooney | rouk: if you just file a bug and state i asked you too then it wont get closed imediatly as a ceph issue :) | |
| 14:11:19 | sean-k-mooney | state something like "nova should timeout request to ceph, instead of hanging on network issues" and descibe the behavior you saw | |
| 14:11:51 | sean-k-mooney | mriedem: i dont know if we can fix it but at least we can sse if its sane or not and then close it :) | |
| 14:12:27 | mriedem | i was just joking | |
| 14:13:39 | rouk | sean-k-mooney: can do. | |
| 14:14:18 | sean-k-mooney | :) i was assuming that but i also need to go grab coffe so brb | |
| 14:14:20 | rouk | oh, launchpad, my ubuntuone cant use launchpad due to "oops" error, no response from support about it... guess ill make a new account or something. | |