| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 10:31:41 | kashyap | pjm6: Yes, that is true _if_ you specifically want the SandyBridge | |
| 10:31:55 | kashyap | pjm6: No, you can't do that | |
| 10:32:15 | pjm6 | kashyap, the problem is that i have two different cpus, one is Broadwell the other is SandyBridge | |
| 10:32:36 | pjm6 | i was trying to "downgrade" the instruction set in order to allow live migration | |
| 10:33:08 | pjm6 | but it seems that when i migrate for the newer to the old, he's comparing physical cpu specs from one server to the other, and then fails | |
| 10:33:14 | pjm6 | 2 line of the first logs | |
| 10:34:05 | kashyap | pjm6: Ah-ha, okay, your problem is clearer. If using the lowest-common denominator CPU model between the two hosts, it should 'just work' | |
| 10:34:49 | pjm6 | yes, that's what gived to me the virsh cpu-baseline (which had the two CPU configs) | |
| 10:38:15 | kashyap | pjm6: That should clearly work. Do you have a libvirt debug log with log filters enabled? And please file a Nova bug with details | |
| 10:38:48 | pjm6 | kashyap, for now i have debug disabled and don't have any filter log | |
| 10:38:56 | pjm6 | but i can activate the debug yes | |
| 10:39:15 | pjm6 | I also see that the dumpxml are using the correct cpu model | |
| 10:40:15 | kashyap | pjm6: BTW, to enable libvirt log filters, this is the way to go: https://kashyapc.fedorapeople.org/virt/openstack/request-nova-libvirt-qemu-debug-logs.txt | |
| 10:40:58 | kashyap | pjm6: What are the libvirt / QEMU versions? Just in case if I happened to have an env. I can test it | |
| 10:41:11 | kashyap | (To see if could reproduce the behaviour.) | |
| 10:41:42 | pjm6 | kashyap, will see that | |
| 10:42:13 | pjm6 | libvirt 2.5.0 | qemu 1:2.8 | |
| 10:43:23 | kashyap | pjm6: Do you have a libvirt guest XML for a VM started on both hosts? | |
| 10:43:36 | kashyap | (Can you post them somewhere, please) | |
| 10:44:38 | pjm6 | kashyap, you want a dumpxml of a VM that was created in compute01 and compute02 right/ | |
| 10:44:53 | kashyap | pjm6: Yep | |
| 10:44:59 | pjm6 | 1 sec please | |
| 10:52:48 | kashyap | pjm6: Also, one more: please get `virsh capabilities` from both hosts | |
| 10:54:55 | pjm6 | dumpxml: https://pastebin.com/xbuA2BFS and https://pastebin.com/NPRAEjbV | |
| 10:57:45 | pjm6 | virsh capabilities: https://pastebin.com/upCBBTu6 and https://pastebin.com/aLDeQipG | |
| 10:58:10 | kashyap | Thanks; will look in a few | |
| 10:58:22 | pjm6 | well kashyap | |
| 10:58:24 | pjm6 | i found a problem! | |
| 10:58:37 | kashyap | pjm6: ? | |
| 10:58:37 | pjm6 | compute node02 for some reason is using broadwell | |
| 10:58:40 | kashyap | Hehe | |
| 10:59:02 | pjm6 | well this is strange | |
| 10:59:06 | pjm6 | first i create in compute01 | |
| 10:59:11 | pjm6 | and then migrate to compute02 | |
| 10:59:20 | pjm6 | the was fine | |
| 10:59:25 | pjm6 | and the dumpxml was using sandybridge | |
| 11:03:21 | kashyap | pjm6: Can you get the real error from /var/log/libvirt/libvirtd.log, too? | |
| 11:04:42 | pjm6 | I don't have that file yet | |
| 11:07:34 | sfinucan | (My) life would be so much easier if every instance had a NUMA topology associated with it | |
| 11:10:02 | pjm6 | kashyap, i already activate as you have | |
| 11:10:20 | pjm6 | in the doc | |
| 11:11:33 | kashyap | pjm6: For the actual libvirt error, maybe it's redirected to systemd journal, do you see anything in: | |
| 11:11:37 | kashyap | journalctl -u libvirtd -l --since=yesterday -p err | |
| 11:11:51 | kashyap | s/yesterday/today/ | |
| 11:12:10 | pjm6 | yeah i see error | |
| 11:12:13 | pjm6 | but idk if is this | |
| 11:12:26 | kashyap | pjm6: Can you paste what it is | |
| 11:12:29 | kashyap | sfinucan: What is causing grief? | |
| 11:12:48 | pjm6 | sure | |
| 11:12:49 | kashyap | Oh, you tell it in the following comment | |
| 11:13:33 | pjm6 | internal error: unable to execute QEMU command 'query-cpu-definitions': The command query-cpu-definitions has not been found | |
| 11:14:14 | sfinucan | sfinucan: People changing things in patches to fix something, but neglecting to mention that they fixed said thing | |
| 11:14:22 | sfinucan | Whoops, kashyap ^ | |
| 11:14:44 | kashyap | sfinucan: Yeah, I hear you | |
| 11:22:26 | kashyap | pjm6: I think that error comes as part of a series of commands libvirt invocates during some high-level operation like migration. | |
| 11:22:49 | kashyap | But anyway, ensuring your destination Compute node really is also using the custom CPU model should get you going. As you've noticed the problem. | |
| 11:23:19 | pjm6 | yeah for some reason idk why when creating a VM he don't update the XML | |
| 11:29:26 | pjm6 | kashyap, | |
| 11:29:31 | pjm6 | it seems it's working now | |
| 11:29:40 | pjm6 | but the virsh dumpxml don't show anything related to cpu | |
| 11:29:41 | pjm6 | this is normal/ | |
| 11:30:46 | pjm6 | forget it appears, with less info | |
| 11:31:18 | pjm6 | https://pastebin.com/gbC4inS | |
| 11:31:25 | pjm6 | i just restared the service | |
| 11:40:07 | openstackgerrit | Daniel Gonzalez Nothnagel proposed openstack/nova master: WIP: Get disk size of instance in separate process https://review.openstack.org/465653 | |
| 12:36:00 | openstackgerrit | jichenjc proposed openstack/nova master: remove unused policy from policy doc https://review.openstack.org/465505 | |
| 12:42:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Factor out duplicated notification sample data (3) https://review.openstack.org/452820 | |
| 12:47:22 | kashyap | pjm6: Your paste-bin has expired; but good if youve' got it working | |
| 12:47:46 | kashyap | mdbooth: Did you notice the above patch? https://review.openstack.org/465653 : | |
| 12:47:55 | kashyap | "[...] While getting instance disk information from the libvirt domain xml, it calls os.path.getsize on the disk path. If this path is unresponsive, e.g. because it is located on an overloaded NFS mount, the nova-compute service hangs and becomes unresponsive. [...]" | |
| 12:48:28 | kashyap | But they're using a separate process to query for each disk, which could be ... "interesting" if there are, say, 50 disks | |
| 12:59:07 | mlakat | kashyap, yes, but what other methods could you suggest? | |
| 12:59:53 | kashyap | mlakat: Hmm, I saw your comment on the thread. | |
| 13:00:16 | mlakat | I also added some more info to today's meeting: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 13:00:30 | kashyap | And, thanks for the link to that eventlet bug, useful to learn about it | |
| 13:05:03 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 13:19:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove DeviceIsBusy exception https://review.openstack.org/465723 | |
| 13:35:40 | artom | mriedem, heya, thanks for +W'ing the bottom 2 tagged attach patches :) Question for you in https://review.openstack.org/#/c/376021/49 - I'm not sure what you mean by moving error handling lower down, as I don't see a way to do it consistently without handling at least 1 new exception in the API layer | |
| 13:35:40 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 13:36:13 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Remove various deprecated options https://review.openstack.org/465502 | |
| 13:38:30 | mriedem | artom: for example https://review.openstack.org/#/c/391941/42/nova/compute/api.py@3786 | |
| 13:38:45 | mriedem | you're now raising VolumeTaggedAttachToShelvedNotSupported from nova.compute.api.API.attach_volume | |
| 13:39:19 | mriedem | but not handling that exception in the REST API controller code https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/volumes.py#L337 | |
| 13:39:25 | mriedem | not in that patch anyway | |
| 13:39:49 | mriedem | and not handling it in the rest api controller code means it would result in a 500 | |
| 13:39:55 | openstackgerrit | Edu Alcaniz proposed openstack/nova master: follow up https://review.openstack.org/465975 | |
| 13:40:04 | mriedem | i realize you can't get that exception until the microversion because the tag isn't in the schema yet, | |
| 13:40:12 | artom | mriedem, oh, you mean move the error handling to the same *patch* | |
| 13:40:16 | artom | Not the same layer | |
| 13:40:17 | mriedem | but i'd prefer to see the error handling for the rest api done in the same methods that introduce those exceptions | |
| 13:40:20 | mriedem | yes | |
| 13:40:23 | mriedem | same patch | |
| 13:40:35 | artom | OK, sorry | |
| 13:40:36 | mriedem | lower layer in the series of changes is what i meant | |
| 13:41:05 | artom | OK, makes sense, will do that :) | |
| 13:43:58 | mriedem | melwitt: replied on https://review.openstack.org/#/c/465042/ - i think we can just backport, resolve the conflict and note it in the commit message | |
| 13:44:04 | mriedem | i don't think we need 2 patches on master for that | |
| 13:44:38 | cdent | mriedem: I think edleafe is at pycon, jaypipes in china, and i have to go outside to wait for someone to come rescue my car, so I think bauzas (if he's around) will be your go to guy for today's meeting's scheduler update | |
| 13:45:21 | mriedem | cdent: i can run it actually, kid is home sick so i'll be here to do it | |
| 13:45:57 | bauzas | yup I'm there | |