| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 10:59:25 | pjm6 | and the dumpxml was using sandybridge | |
| 11:03:21 | kashyap | pjm6: Can you get the real error from /var/log/libvirt/libvirtd.log, too? | |
| 11:04:42 | pjm6 | I don't have that file yet | |
| 11:07:34 | sfinucan | (My) life would be so much easier if every instance had a NUMA topology associated with it | |
| 11:10:02 | pjm6 | kashyap, i already activate as you have | |
| 11:10:20 | pjm6 | in the doc | |
| 11:11:33 | kashyap | pjm6: For the actual libvirt error, maybe it's redirected to systemd journal, do you see anything in: | |
| 11:11:37 | kashyap | journalctl -u libvirtd -l --since=yesterday -p err | |
| 11:11:51 | kashyap | s/yesterday/today/ | |
| 11:12:10 | pjm6 | yeah i see error | |
| 11:12:13 | pjm6 | but idk if is this | |
| 11:12:26 | kashyap | pjm6: Can you paste what it is | |
| 11:12:29 | kashyap | sfinucan: What is causing grief? | |
| 11:12:48 | pjm6 | sure | |
| 11:12:49 | kashyap | Oh, you tell it in the following comment | |
| 11:13:33 | pjm6 | internal error: unable to execute QEMU command 'query-cpu-definitions': The command query-cpu-definitions has not been found | |
| 11:14:14 | sfinucan | sfinucan: People changing things in patches to fix something, but neglecting to mention that they fixed said thing | |
| 11:14:22 | sfinucan | Whoops, kashyap ^ | |
| 11:14:44 | kashyap | sfinucan: Yeah, I hear you | |
| 11:22:26 | kashyap | pjm6: I think that error comes as part of a series of commands libvirt invocates during some high-level operation like migration. | |
| 11:22:49 | kashyap | But anyway, ensuring your destination Compute node really is also using the custom CPU model should get you going. As you've noticed the problem. | |
| 11:23:19 | pjm6 | yeah for some reason idk why when creating a VM he don't update the XML | |
| 11:29:26 | pjm6 | kashyap, | |
| 11:29:31 | pjm6 | it seems it's working now | |
| 11:29:40 | pjm6 | but the virsh dumpxml don't show anything related to cpu | |
| 11:29:41 | pjm6 | this is normal/ | |
| 11:30:46 | pjm6 | forget it appears, with less info | |
| 11:31:18 | pjm6 | https://pastebin.com/gbC4inS | |
| 11:31:25 | pjm6 | i just restared the service | |
| 11:40:07 | openstackgerrit | Daniel Gonzalez Nothnagel proposed openstack/nova master: WIP: Get disk size of instance in separate process https://review.openstack.org/465653 | |
| 12:36:00 | openstackgerrit | jichenjc proposed openstack/nova master: remove unused policy from policy doc https://review.openstack.org/465505 | |
| 12:42:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Factor out duplicated notification sample data (3) https://review.openstack.org/452820 | |
| 12:47:22 | kashyap | pjm6: Your paste-bin has expired; but good if youve' got it working | |
| 12:47:46 | kashyap | mdbooth: Did you notice the above patch? https://review.openstack.org/465653 : | |
| 12:47:55 | kashyap | "[...] While getting instance disk information from the libvirt domain xml, it calls os.path.getsize on the disk path. If this path is unresponsive, e.g. because it is located on an overloaded NFS mount, the nova-compute service hangs and becomes unresponsive. [...]" | |
| 12:48:28 | kashyap | But they're using a separate process to query for each disk, which could be ... "interesting" if there are, say, 50 disks | |
| 12:59:07 | mlakat | kashyap, yes, but what other methods could you suggest? | |
| 12:59:53 | kashyap | mlakat: Hmm, I saw your comment on the thread. | |
| 13:00:16 | mlakat | I also added some more info to today's meeting: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 13:00:30 | kashyap | And, thanks for the link to that eventlet bug, useful to learn about it | |
| 13:05:03 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 13:19:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove DeviceIsBusy exception https://review.openstack.org/465723 | |
| 13:35:40 | artom | mriedem, heya, thanks for +W'ing the bottom 2 tagged attach patches :) Question for you in https://review.openstack.org/#/c/376021/49 - I'm not sure what you mean by moving error handling lower down, as I don't see a way to do it consistently without handling at least 1 new exception in the API layer | |
| 13:35:40 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 13:36:13 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Remove various deprecated options https://review.openstack.org/465502 | |
| 13:38:30 | mriedem | artom: for example https://review.openstack.org/#/c/391941/42/nova/compute/api.py@3786 | |
| 13:38:45 | mriedem | you're now raising VolumeTaggedAttachToShelvedNotSupported from nova.compute.api.API.attach_volume | |
| 13:39:19 | mriedem | but not handling that exception in the REST API controller code https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/volumes.py#L337 | |
| 13:39:25 | mriedem | not in that patch anyway | |
| 13:39:49 | mriedem | and not handling it in the rest api controller code means it would result in a 500 | |
| 13:39:55 | openstackgerrit | Edu Alcaniz proposed openstack/nova master: follow up https://review.openstack.org/465975 | |
| 13:40:04 | mriedem | i realize you can't get that exception until the microversion because the tag isn't in the schema yet, | |
| 13:40:12 | artom | mriedem, oh, you mean move the error handling to the same *patch* | |
| 13:40:16 | artom | Not the same layer | |
| 13:40:17 | mriedem | but i'd prefer to see the error handling for the rest api done in the same methods that introduce those exceptions | |
| 13:40:20 | mriedem | yes | |
| 13:40:23 | mriedem | same patch | |
| 13:40:35 | artom | OK, sorry | |
| 13:40:36 | mriedem | lower layer in the series of changes is what i meant | |
| 13:41:05 | artom | OK, makes sense, will do that :) | |
| 13:43:58 | mriedem | melwitt: replied on https://review.openstack.org/#/c/465042/ - i think we can just backport, resolve the conflict and note it in the commit message | |
| 13:44:04 | mriedem | i don't think we need 2 patches on master for that | |
| 13:44:38 | cdent | mriedem: I think edleafe is at pycon, jaypipes in china, and i have to go outside to wait for someone to come rescue my car, so I think bauzas (if he's around) will be your go to guy for today's meeting's scheduler update | |
| 13:45:21 | mriedem | cdent: i can run it actually, kid is home sick so i'll be here to do it | |
| 13:45:57 | bauzas | yup I'm there | |
| 13:46:12 | cdent | i assumed you were running it, mriedem, just the scheduler part, which looks like bauzas has under control | |
| 13:46:24 | mriedem | oh i see | |
| 13:46:25 | mriedem | ok | |
| 13:47:00 | bauzas | fine by me | |
| 13:57:07 | mriedem | grenade live migration job is basically 100% fail http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/console.html#_2017-05-18_00_39_25_007223 | |
| 13:57:13 | mriedem | Failed to restart devstack@g-api.service: Unit devstack@g-api.service not found. | |
| 13:57:50 | mriedem | i assume because on master that code is expecting systemd, but started on ocata with screen | |
| 14:00:49 | sdague | yeh, though that should have been pinned | |
| 14:01:26 | sdague | mriedem: oh, the live migration scripts are all super custom | |
| 14:01:33 | sdague | they are going to need adjustments | |
| 14:02:27 | sdague | mriedem: grenade completed fine | |
| 14:02:46 | sdague | the scripts after that are making assumptions | |
| 14:03:34 | mriedem | sdague: will dig after the nova meeting | |
| 14:15:29 | openstack | Launchpad bug 1691769 in OpenStack Compute (nova) "gate-grenade-dsvm-neutron-multinode-live-migration-nv fails in pike: "Failed to restart <email address hidden>: Unit <email address hidden> not found."" [Medium,Triaged] | |
| 14:15:29 | mriedem | https://bugs.launchpad.net/nova/+bug/1691769 | |
| 14:40:30 | takashin | dansmith: Would you review https://review.openstack.org/#/c/408955/ again? | |
| 14:44:41 | mriedem | takashin: i'm confused, | |
| 14:44:51 | mriedem | https://review.openstack.org/#/c/408964/49/nova/api/openstack/api_version_request.py and https://review.openstack.org/#/c/459483/3/nova/api/openstack/api_version_request.py are competing for the same microversion, | |
| 14:45:05 | mriedem | but i thought the abort cold migration blueprint dependent on being able to list cold migrations? | |
| 14:45:26 | mriedem | oh nvm | |
| 14:45:30 | mriedem | i'm thinking of a different blueprint | |
| 14:45:37 | mriedem | this is cold migration with a target host | |
| 14:51:25 | takashin | mriedem: As you said, there is no dependency between "cold migration with target host" and "List/show all migration types" blueprint. | |
| 14:55:35 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix uuid replacement in aggregate notification test https://review.openstack.org/465896 | |
| 14:55:42 | mriedem | sdague: i'm confused that USE_SCREEN=False on the old side of this live migration grenade job http://logs.openstack.org/42/465042/5/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/6aae0c3/logs/old/local_conf.txt.gz | |
| 14:55:43 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix uuid replacement in aggregate notification test https://review.openstack.org/465896 | |
| 14:55:48 | mriedem | i thought screen was still the default in devstack in ocata? | |
| 14:56:43 | mriedem | i guess it is https://github.com/openstack-dev/devstack/blob/stable/ocata/stackrc#L88 | |
| 14:58:24 | sdague | mriedem: the confusion is there are 3 operating modes right now | |
| 14:59:10 | sdague | USE_SCREEN=False before a month ago, means direct exec everything in bash and hope you captured the pid correctly to kill it later | |
| 14:59:32 | sdague | we've used that in the gate for the last 3 years because screen races | |
| 14:59:54 | sdague | and in the non repeat state, it's fine. | |
| 15:00:21 | sdague | it is a different home grown init system with different bad trade offs than screen | |
| 15:01:58 | dpawlik | Hi.If you have some time pls review my PS https://review.openstack.org/#/c/459753/ | |
| 15:02:00 | dpawlik | thanks a lot | |