| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 10:44:53 | kashyap | pjm6: Yep | |
| 10:44:59 | pjm6 | 1 sec please | |
| 10:52:48 | kashyap | pjm6: Also, one more: please get `virsh capabilities` from both hosts | |
| 10:54:55 | pjm6 | dumpxml: https://pastebin.com/xbuA2BFS and https://pastebin.com/NPRAEjbV | |
| 10:57:45 | pjm6 | virsh capabilities: https://pastebin.com/upCBBTu6 and https://pastebin.com/aLDeQipG | |
| 10:58:10 | kashyap | Thanks; will look in a few | |
| 10:58:22 | pjm6 | well kashyap | |
| 10:58:24 | pjm6 | i found a problem! | |
| 10:58:37 | kashyap | pjm6: ? | |
| 10:58:37 | pjm6 | compute node02 for some reason is using broadwell | |
| 10:58:40 | kashyap | Hehe | |
| 10:59:02 | pjm6 | well this is strange | |
| 10:59:06 | pjm6 | first i create in compute01 | |
| 10:59:11 | pjm6 | and then migrate to compute02 | |
| 10:59:20 | pjm6 | the was fine | |
| 10:59:25 | pjm6 | and the dumpxml was using sandybridge | |
| 11:03:21 | kashyap | pjm6: Can you get the real error from /var/log/libvirt/libvirtd.log, too? | |
| 11:04:42 | pjm6 | I don't have that file yet | |
| 11:07:34 | sfinucan | (My) life would be so much easier if every instance had a NUMA topology associated with it | |
| 11:10:02 | pjm6 | kashyap, i already activate as you have | |
| 11:10:20 | pjm6 | in the doc | |
| 11:11:33 | kashyap | pjm6: For the actual libvirt error, maybe it's redirected to systemd journal, do you see anything in: | |
| 11:11:37 | kashyap | journalctl -u libvirtd -l --since=yesterday -p err | |
| 11:11:51 | kashyap | s/yesterday/today/ | |
| 11:12:10 | pjm6 | yeah i see error | |
| 11:12:13 | pjm6 | but idk if is this | |
| 11:12:26 | kashyap | pjm6: Can you paste what it is | |
| 11:12:29 | kashyap | sfinucan: What is causing grief? | |
| 11:12:48 | pjm6 | sure | |
| 11:12:49 | kashyap | Oh, you tell it in the following comment | |
| 11:13:33 | pjm6 | internal error: unable to execute QEMU command 'query-cpu-definitions': The command query-cpu-definitions has not been found | |
| 11:14:14 | sfinucan | sfinucan: People changing things in patches to fix something, but neglecting to mention that they fixed said thing | |
| 11:14:22 | sfinucan | Whoops, kashyap ^ | |
| 11:14:44 | kashyap | sfinucan: Yeah, I hear you | |
| 11:22:26 | kashyap | pjm6: I think that error comes as part of a series of commands libvirt invocates during some high-level operation like migration. | |
| 11:22:49 | kashyap | But anyway, ensuring your destination Compute node really is also using the custom CPU model should get you going. As you've noticed the problem. | |
| 11:23:19 | pjm6 | yeah for some reason idk why when creating a VM he don't update the XML | |
| 11:29:26 | pjm6 | kashyap, | |
| 11:29:31 | pjm6 | it seems it's working now | |
| 11:29:40 | pjm6 | but the virsh dumpxml don't show anything related to cpu | |
| 11:29:41 | pjm6 | this is normal/ | |
| 11:30:46 | pjm6 | forget it appears, with less info | |
| 11:31:18 | pjm6 | https://pastebin.com/gbC4inS | |
| 11:31:25 | pjm6 | i just restared the service | |
| 11:40:07 | openstackgerrit | Daniel Gonzalez Nothnagel proposed openstack/nova master: WIP: Get disk size of instance in separate process https://review.openstack.org/465653 | |
| 12:36:00 | openstackgerrit | jichenjc proposed openstack/nova master: remove unused policy from policy doc https://review.openstack.org/465505 | |
| 12:42:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Factor out duplicated notification sample data (3) https://review.openstack.org/452820 | |
| 12:47:22 | kashyap | pjm6: Your paste-bin has expired; but good if youve' got it working | |
| 12:47:46 | kashyap | mdbooth: Did you notice the above patch? https://review.openstack.org/465653 : | |
| 12:47:55 | kashyap | "[...] While getting instance disk information from the libvirt domain xml, it calls os.path.getsize on the disk path. If this path is unresponsive, e.g. because it is located on an overloaded NFS mount, the nova-compute service hangs and becomes unresponsive. [...]" | |
| 12:48:28 | kashyap | But they're using a separate process to query for each disk, which could be ... "interesting" if there are, say, 50 disks | |
| 12:59:07 | mlakat | kashyap, yes, but what other methods could you suggest? | |
| 12:59:53 | kashyap | mlakat: Hmm, I saw your comment on the thread. | |
| 13:00:16 | mlakat | I also added some more info to today's meeting: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 13:00:30 | kashyap | And, thanks for the link to that eventlet bug, useful to learn about it | |
| 13:05:03 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 13:19:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove DeviceIsBusy exception https://review.openstack.org/465723 | |
| 13:35:40 | artom | mriedem, heya, thanks for +W'ing the bottom 2 tagged attach patches :) Question for you in https://review.openstack.org/#/c/376021/49 - I'm not sure what you mean by moving error handling lower down, as I don't see a way to do it consistently without handling at least 1 new exception in the API layer | |
| 13:35:40 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 13:36:13 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Remove various deprecated options https://review.openstack.org/465502 | |
| 13:38:30 | mriedem | artom: for example https://review.openstack.org/#/c/391941/42/nova/compute/api.py@3786 | |
| 13:38:45 | mriedem | you're now raising VolumeTaggedAttachToShelvedNotSupported from nova.compute.api.API.attach_volume | |
| 13:39:19 | mriedem | but not handling that exception in the REST API controller code https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/volumes.py#L337 | |
| 13:39:25 | mriedem | not in that patch anyway | |
| 13:39:49 | mriedem | and not handling it in the rest api controller code means it would result in a 500 | |
| 13:39:55 | openstackgerrit | Edu Alcaniz proposed openstack/nova master: follow up https://review.openstack.org/465975 | |
| 13:40:04 | mriedem | i realize you can't get that exception until the microversion because the tag isn't in the schema yet, | |
| 13:40:12 | artom | mriedem, oh, you mean move the error handling to the same *patch* | |
| 13:40:16 | artom | Not the same layer | |
| 13:40:17 | mriedem | but i'd prefer to see the error handling for the rest api done in the same methods that introduce those exceptions | |
| 13:40:20 | mriedem | yes | |
| 13:40:23 | mriedem | same patch | |
| 13:40:35 | artom | OK, sorry | |
| 13:40:36 | mriedem | lower layer in the series of changes is what i meant | |
| 13:41:05 | artom | OK, makes sense, will do that :) | |
| 13:43:58 | mriedem | melwitt: replied on https://review.openstack.org/#/c/465042/ - i think we can just backport, resolve the conflict and note it in the commit message | |
| 13:44:04 | mriedem | i don't think we need 2 patches on master for that | |
| 13:44:38 | cdent | mriedem: I think edleafe is at pycon, jaypipes in china, and i have to go outside to wait for someone to come rescue my car, so I think bauzas (if he's around) will be your go to guy for today's meeting's scheduler update | |
| 13:45:21 | mriedem | cdent: i can run it actually, kid is home sick so i'll be here to do it | |
| 13:45:57 | bauzas | yup I'm there | |
| 13:46:12 | cdent | i assumed you were running it, mriedem, just the scheduler part, which looks like bauzas has under control | |
| 13:46:24 | mriedem | oh i see | |
| 13:46:25 | mriedem | ok | |
| 13:47:00 | bauzas | fine by me | |
| 13:57:07 | mriedem | grenade live migration job is basically 100% fail http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/console.html#_2017-05-18_00_39_25_007223 | |
| 13:57:13 | mriedem | Failed to restart devstack@g-api.service: Unit devstack@g-api.service not found. | |
| 13:57:50 | mriedem | i assume because on master that code is expecting systemd, but started on ocata with screen | |
| 14:00:49 | sdague | yeh, though that should have been pinned | |
| 14:01:26 | sdague | mriedem: oh, the live migration scripts are all super custom | |
| 14:01:33 | sdague | they are going to need adjustments | |
| 14:02:27 | sdague | mriedem: grenade completed fine | |
| 14:02:46 | sdague | the scripts after that are making assumptions | |
| 14:03:34 | mriedem | sdague: will dig after the nova meeting | |
| 14:15:29 | openstack | Launchpad bug 1691769 in OpenStack Compute (nova) "gate-grenade-dsvm-neutron-multinode-live-migration-nv fails in pike: "Failed to restart <email address hidden>: Unit <email address hidden> not found."" [Medium,Triaged] | |
| 14:15:29 | mriedem | https://bugs.launchpad.net/nova/+bug/1691769 | |
| 14:40:30 | takashin | dansmith: Would you review https://review.openstack.org/#/c/408955/ again? | |
| 14:44:41 | mriedem | takashin: i'm confused, | |
| 14:44:51 | mriedem | https://review.openstack.org/#/c/408964/49/nova/api/openstack/api_version_request.py and https://review.openstack.org/#/c/459483/3/nova/api/openstack/api_version_request.py are competing for the same microversion, | |
| 14:45:05 | mriedem | but i thought the abort cold migration blueprint dependent on being able to list cold migrations? | |
| 14:45:26 | mriedem | oh nvm | |