Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-18
09:46:55 kashyap mdbooth: Noticed the other day, wow, this is merged: https://review.openstack.org/#/c/383859/ ("Fix races with nfs volume mount/umount")
09:47:07 mdbooth kashyap: Yeah :)
10:13:32 pjm6 good morning
10:13:35 pjm6 i have a problem with live migration, basically I put a cpu_mode: custom and cpu_model: SandyBridge
10:13:38 pjm6 when i migrate from compute01 to compute02 it works good
10:13:40 pjm6 but the other way arround no, because tthe compute02 has more flags
10:13:43 pjm6 but by forcing the cpu_model SandyBridge this should be working, no?
10:15:08 kashyap pjm6: Do you have the same cpu_{mode,model} config on all Compute nodes
10:16:09 pjm6 kashyap, yes I have
10:16:40 kashyap BTW, what release are you trying this with? Can't say more without details
10:16:51 pjm6 of course, sorry
10:16:56 pjm6 i'm in Ocata version
10:17:09 pjm6 using nova 15.1.2
10:20:13 pjm6 kashyap, this is old but https://blog.codecentric.de/en/2015/03/true-kvm-live-migration-openstack-icehouse-ceph-based-vm-storage/ it seems that I have the problem #2
10:23:26 pjm6 the error says: Unacceptable CPU info: CPU doesn't have compatibility
10:24:59 kashyap pjm6: That was in IceHouse
10:25:26 pjm6 kashyap, yes I said that was an older version, but the problem seems similar
10:25:28 pjm6 for isntance
10:26:20 pjm6 the error log: https://pastebin.com/zd35WqqJ
10:27:22 pjm6 then I use the virsh cpu-baseline and I get this: https://pastebin.com/AxUx0kYf
10:28:06 kashyap pjm6: BTW, just a gentle note: this channel is for devel purposes, maybe you're already aware
10:30:21 kashyap pjm6: I'm just curious, any reason you don't use 'host-model'?
10:30:39 kashyap Which is actually the default, which use the closest available named-CPU model
10:31:14 pjm6 kashyap, i didn't use host-model because I though that if i wanted to use SandyBridge the cpu_mode should be custom
10:31:36 pjm6 or i can use cpu_mode: host-model and cpu_model: SandyBridge?
10:31:41 kashyap pjm6: Yes, that is true _if_ you specifically want the SandyBridge
10:31:55 kashyap pjm6: No, you can't do that
10:32:15 pjm6 kashyap, the problem is that i have two different cpus, one is Broadwell the other is SandyBridge
10:32:36 pjm6 i was trying to "downgrade" the instruction set in order to allow live migration
10:33:08 pjm6 but it seems that when i migrate for the newer to the old, he's comparing physical cpu specs from one server to the other, and then fails
10:33:14 pjm6 2 line of the first logs
10:34:05 kashyap pjm6: Ah-ha, okay, your problem is clearer. If using the lowest-common denominator CPU model between the two hosts, it should 'just work'
10:34:49 pjm6 yes, that's what gived to me the virsh cpu-baseline (which had the two CPU configs)
10:38:15 kashyap pjm6: That should clearly work. Do you have a libvirt debug log with log filters enabled? And please file a Nova bug with details
10:38:48 pjm6 kashyap, for now i have debug disabled and don't have any filter log
10:38:56 pjm6 but i can activate the debug yes
10:39:15 pjm6 I also see that the dumpxml are using the correct cpu model
10:40:15 kashyap pjm6: BTW, to enable libvirt log filters, this is the way to go: https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt
10:40:58 kashyap pjm6: What are the libvirt / QEMU versions? Just in case if I happened to have an env. I can test it
10:41:11 kashyap (To see if could reproduce the behaviour.)
10:41:42 pjm6 kashyap, will see that
10:42:13 pjm6 libvirt 2.5.0 | qemu 1:2.8
10:43:23 kashyap pjm6: Do you have a libvirt guest XML for a VM started on both hosts?
10:43:36 kashyap (Can you post them somewhere, please)
10:44:38 pjm6 kashyap, you want a dumpxml of a VM that was created in compute01 and compute02 right/
10:44:53 kashyap pjm6: Yep
10:44:59 pjm6 1 sec please
10:52:48 kashyap pjm6: Also, one more: please get `virsh capabilities` from both hosts
10:54:55 pjm6 dumpxml: https://pastebin.com/xbuA2BFS and https://pastebin.com/NPRAEjbV
10:57:45 pjm6 virsh capabilities: https://pastebin.com/upCBBTu6 and https://pastebin.com/aLDeQipG
10:58:10 kashyap Thanks; will look in a few
10:58:22 pjm6 well kashyap
10:58:24 pjm6 i found a problem!
10:58:37 kashyap pjm6: ?
10:58:37 pjm6 compute node02 for some reason is using broadwell
10:58:40 kashyap Hehe
10:59:02 pjm6 well this is strange
10:59:06 pjm6 first i create in compute01
10:59:11 pjm6 and then migrate to compute02
10:59:20 pjm6 the was fine
10:59:25 pjm6 and the dumpxml was using sandybridge
11:03:21 kashyap pjm6: Can you get the real error from /var/log/libvirt/libvirtd.log, too?
11:04:42 pjm6 I don't have that file yet
11:07:34 sfinucan (My) life would be so much easier if every instance had a NUMA topology associated with it
11:10:02 pjm6 kashyap, i already activate as you have
11:10:20 pjm6 in the doc
11:11:33 kashyap pjm6: For the actual libvirt error, maybe it's redirected to systemd journal, do you see anything in:
11:11:37 kashyap journalctl -u libvirtd -l --since=yesterday -p err
11:11:51 kashyap s/yesterday/today/
11:12:10 pjm6 yeah i see error
11:12:13 pjm6 but idk if is this
11:12:26 kashyap pjm6: Can you paste what it is
11:12:29 kashyap sfinucan: What is causing grief?
11:12:48 pjm6 sure
11:12:49 kashyap Oh, you tell it in the following comment
11:13:33 pjm6 internal error: unable to execute QEMU command 'query-cpu-definitions': The command query-cpu-definitions has not been found
11:14:14 sfinucan sfinucan: People changing things in patches to fix something, but neglecting to mention that they fixed said thing
11:14:22 sfinucan Whoops, kashyap ^
11:14:44 kashyap sfinucan: Yeah, I hear you
11:22:26 kashyap pjm6: I think that error comes as part of a series of commands libvirt invocates during some high-level operation like migration.
11:22:49 kashyap But anyway, ensuring your destination Compute node really is also using the custom CPU model should get you going. As you've noticed the problem.
11:23:19 pjm6 yeah for some reason idk why when creating a VM he don't update the XML
11:29:26 pjm6 kashyap,
11:29:31 pjm6 it seems it's working now
11:29:40 pjm6 but the virsh dumpxml don't show anything related to cpu
11:29:41 pjm6 this is normal/
11:30:46 pjm6 forget it appears, with less info
11:31:18 pjm6 https://pastebin.com/gbC4inS
11:31:25 pjm6 i just restared the service
11:40:07 openstackgerrit Daniel Gonzalez Nothnagel proposed openstack/nova master: WIP: Get disk size of instance in separate process https://review.openstack.org/465653
12:36:00 openstackgerrit jichenjc proposed openstack/nova master: remove unused policy from policy doc https://review.openstack.org/465505
12:42:21 openstackgerrit Balazs Gibizer proposed openstack/nova master: Factor out duplicated notification sample data (3) https://review.openstack.org/452820
12:47:22 kashyap pjm6: Your paste-bin has expired; but good if youve' got it working
12:47:46 kashyap mdbooth: Did you notice the above patch? https://review.openstack.org/465653 :
12:47:55 kashyap "[...] While getting instance disk information from the libvirt domain xml, it calls os.path.getsize on the disk path. If this path is unresponsive, e.g. because it is located on an overloaded NFS mount, the nova-compute service hangs and becomes unresponsive. [...]"
12:48:28 kashyap But they're using a separate process to query for each disk, which could be ... "interesting" if there are, say, 50 disks
12:59:07 mlakat kashyap, yes, but what other methods could you suggest?
12:59:53 kashyap mlakat: Hmm, I saw your comment on the thread.
13:00:16 mlakat I also added some more info to today's meeting: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
13:00:30 kashyap And, thanks for the link to that eventlet bug, useful to learn about it

Earlier   Later