| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-14 | |||
| 14:13:42 | sean-k-mooney | ya i have seen that before. | |
| 14:13:54 | sean-k-mooney | are you deploying on nfs or shared stoage | |
| 14:14:04 | dansmith | luyao: no, you can't access the database directly, but the flavor is on the instance so it should already be there and have what you need | |
| 14:15:15 | sean-k-mooney | spatel: this can happen when an instance was delete in the db when the compute agent is stopped and then was archived before it was started again | |
| 14:15:50 | sean-k-mooney | e.g. after an evacutation | |
| 14:16:10 | luyao | dansmith: https://review.opendev.org/#/c/672957/5/nova/virt/libvirt/device.py@196 I need to access db to get instance | |
| 14:16:35 | sean-k-mooney | the host has a stale libvirt domain that references a disk tha that is nolonger found | |
| 14:16:39 | dansmith | luyao: you've got it right there | |
| 14:16:43 | spatel | sean-k-mooney: you are saying someone delete instance but compute agent was down and it got out of sync. | |
| 14:17:29 | dansmith | lyarwood: Instance.get_by_uuid() is okay, it accesses the database, but not directly, through conductor | |
| 14:17:34 | sean-k-mooney | yes basically this happens becasue a statle libvirt domain xml is on the host that references an image file that nolonger exists | |
| 14:17:38 | dansmith | oops | |
| 14:17:39 | spatel | I believe this is related to resize issue when you do CPU pinning which left resize stall | |
| 14:17:43 | dansmith | luyao: ^ | |
| 14:18:13 | luyao | dansmith: so it's ok to access database like this? I just thought it's not recommended to access database in driver | |
| 14:18:15 | dansmith | luyao: but you should try to avoid that if you can pull the instance there.. I'd have to look at the code to see where this is running | |
| 14:18:35 | dansmith | luyao: like I just said, you're not hitting the database directly with that call | |
| 14:19:19 | sean-k-mooney | spatel: it does look similar to https://bugs.launchpad.net/nova/+bug/1774249 | |
| 14:19:19 | openstack | Launchpad bug 1774249 in OpenStack Compute (nova) pike "update_available_resource will raise DiskNotFound after resize but before confirm" [Medium,Triaged] | |
| 14:19:22 | spatel | sean-k-mooney: do you know how do i fix this issue or clean up left over disk. | |
| 14:20:01 | dansmith | luyao: see there are now two places in that init routine that get instances, the mdev one and your pmem one | |
| 14:20:29 | dansmith | luyao: the mdev one is more efficient than what you have, but it's not okay to look up all the instances twice, so you need to refactor that code in a patch ahead of time, | |
| 14:20:31 | alex_xu | dansmith: yea, that code is running when the nova-compute start, we populate all the assigned vgpu or vpmems from the libvirt, then we need instance_uuid and flavor_id to identify each assignment | |
| 14:20:33 | sean-k-mooney | well the disk not found error means there is no left over disk | |
| 14:20:47 | dansmith | luyao: to look up the instances on the driver once and pass them to the mdev and pmem methods | |
| 14:20:47 | sean-k-mooney | spatel: you have a left over domain xml | |
| 14:20:55 | spatel | let me check | |
| 14:21:10 | dansmith | luyao: but ideally you would pass all that in from the actual driver init, because it too does this lookup I think | |
| 14:21:35 | sean-k-mooney | so you can delete it with virsh but you should first check what the instance uuid was and then check tha tnova does not thinkg that vm should be running on that host | |
| 14:22:30 | sean-k-mooney | if nova thinkgs the vm should be delete or running no a different host then provide its not in the resize_confirm state its safe to delete teh domain xml | |
| 14:24:40 | spatel | sean-k-mooney: i think i have xml file in /etc/libvirt/qemu which is not active in virsh list | |
| 14:24:54 | luyao | dansmith: Does driver init lookup all instance objects? I think this should be done in compute manager. | |
| 14:25:11 | dansmith | luyao: I said "I think", I'll look | |
| 14:25:48 | alex_xu | I guess not | |
| 14:25:55 | spatel | sean-k-mooney: can i just delete that file? | |
| 14:26:28 | alex_xu | now I think it is not | |
| 14:26:57 | mriedem | gibi: some thougths in your tempest change to run resize+confirm in grenade https://review.opendev.org/#/c/675371/ | |
| 14:27:05 | alex_xu | dansmith: is it ok query all the related instance objects and pass into the driver.init_host? | |
| 14:27:26 | dansmith | alex_xu: luyao it does indirectly because compute manager calls init_instance for each, but yeah not in driver init, although I'm not sure where we get devicemanager instantiated either | |
| 14:28:34 | dansmith | alex_xu: I would think refactoring init_host to take/pass all instances might be good, yeah | |
| 14:28:39 | dansmith | let me look at something else | |
| 14:29:22 | alex_xu | luyao: ^ is that doable, I guess you need all the instance which instance.host = self.host and something instance in resizing? | |
| 14:29:47 | sean-k-mooney | mriedem: regarding the lxc issue looking at https://logs.opendev.org/24/676024/5/experimental/nova-lxc/9c06394/controller/logs/screen-n-cpu.txt.gz#_Aug_13_23_16_20_055037 we are setting the container entrypoint to /sbin/init but that can be changed in the image metadata https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L63-L67 | |
| 14:30:27 | dansmith | alex_xu: luyao I was going to say, init_instance() already gets called for each instance in compute manager, and you could put your accounting code in there, but passing that list to init_host() would be just as good, if that works | |
| 14:30:33 | sean-k-mooney | so i think for the debian issue we just need to set os_command_line in the lxc image to the path to the init system with is likely systemd now | |
| 14:30:40 | luyao | alex_xu: I need all instances which have xml on the host. | |
| 14:30:42 | dansmith | that would *improve* the current code, instead of your code making it *worse* | |
| 14:31:52 | alex_xu | luyao: is there a case, you have xml for the target instance when resize, but the instance.host is still the src host? | |
| 14:32:11 | Tianhao_Hu | @mriedem hi matt, about this issue, if the directory left after cold migration is empty and has no effect on cold migration, can you gvie some advice about whether we can think this is not a bug? | |
| 14:32:17 | Tianhao_Hu | https://bugs.launchpad.net/starlingx/+bug/1824858 | |
| 14:32:17 | openstack | Launchpad bug 1824858 in StarlingX "nova instance remnant left behind after cold migration completes" [Low,Confirmed] - Assigned to hutianhao27 (hutianhao) | |
| 14:32:55 | alex_xu | dansmith: yea, that is better, just need to check with luyao for the resize case, in case we have domain xml but the instance.host isn't the current host. I guess init won't get the instance doesn't belong to this host | |
| 14:32:59 | luyao | alex_xu: there is a case, the resizing is not finished ,but the instance.host is dest host | |
| 14:33:11 | dansmith | alex_xu: yeah | |
| 14:33:34 | sean-k-mooney | mriedem: could we use a different image in the ci job or set the path to the correct entrypoint. ill look at this again later but i dont think this is a nova bug persay just incorrect configuration | |
| 14:34:23 | alex_xu | luyao: ah, so the source host can get a domain xml, but it need the instance which host is target host | |
| 14:35:08 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.76: Add 'power-update' external event https://review.opendev.org/645611 | |
| 14:35:23 | luyao | alex_xu,: yes , so we may also need check migrations | |
| 14:36:01 | alex_xu | dansmith: I think I can query more instances like the case luyao said, those instances only pass to init_host, but won't be used for later compute-manage initialize | |
| 14:36:21 | mriedem | Tianhao_Hu: i can't say off the top of my head and without digging through the 23 comments and description and recreate of that bug, which i'm unable to do right now | |
| 14:36:39 | mriedem | it also looks like it's rbd backend specific, and i don't have an environment to poke into that or recreate it right now | |
| 14:36:53 | mriedem | someone from starlingx that is familiar with the nova code could triage it | |
| 14:36:59 | mriedem | could/should | |
| 14:38:03 | dansmith | alex_xu: if you pass the list of instances on this host to init_host(), then you can get any instances that have xml but are not in that list, and do an independent query for those, which will be much smaller | |
| 14:38:40 | mriedem | sean-k-mooney: i agree it's some kind of misconfig and related to needing to tell the image to use systemd now | |
| 14:38:45 | dansmith | alex_xu: however, I would think that if you're just trying to figure out which namespaces are assigned, the xml should be enough, so I'm not sure why you need the instance from the db (but I haven't looked closely) | |
| 14:40:04 | alex_xu | dansmith: we need the flavor id. since we use (instance_uuid, flavor_id) to identify a claim. This is due to the same host resize. Both src and dest claim on the same host, we need a way to distinguish the claim | |
| 14:40:46 | alex_xu | the instance uuid I can get from the domain xml. the only trouble is the flavor id | |
| 14:41:39 | sean-k-mooney | mriedem: looking at the lxc buster(debian 10) images the symlink appear to be there. /sbin/init -> /lib/systemd/systemd | |
| 14:41:55 | sean-k-mooney | mriedem: so i think if we just use the right image it should just work | |
| 14:42:26 | mriedem | alex_xu: the flavor name is in the domain xml metadata | |
| 14:42:31 | mriedem | see https://logs.opendev.org/24/676024/5/experimental/nova-lxc/9c06394/controller/logs/screen-n-cpu.txt.gz#_Aug_13_23_16_20_055037 | |
| 14:42:39 | mriedem | <nova:flavor name="m1.nano"> | |
| 14:42:43 | dansmith | alex_xu: okay makes sense | |
| 14:42:56 | alex_xu | mriedem: can we ensure all the virt driver persistent the flavor name? or it is libvirt specific. | |
| 14:43:09 | mriedem | idk about the other drivers | |
| 14:43:13 | mriedem | but they aren't doing pmems either | |
| 14:43:16 | dansmith | alex_xu: mriedem you need more than that, you need the *actual* details of the flavor right? | |
| 14:43:18 | sean-k-mooney | alex_xu: i think the metadata is driver specific | |
| 14:43:30 | sean-k-mooney | alex_xu: it normally has the flavor uuid too | |
| 14:43:32 | dansmith | alex_xu: looking up the flavor by name after the instance is spawned won't work because it may have changed, you need instance.flavor I think | |
| 14:43:38 | mriedem | i'm wading into a conversation for which i don't have context of the problem, i'm just saying, the domain xml has flavor info | |
| 14:43:40 | alex_xu | dansmith: only need the id | |
| 14:44:01 | dansmith | alex_xu: what does that tell you? | |
| 14:44:35 | alex_xu | dansmith: I just want to distinguish the src claim and dest claim on the same host. so instance_uuid and flavor_id is enough | |
| 14:44:38 | mriedem | the domain xml metadata doesn't store the flavor id so i guess that won't help you | |
| 14:44:46 | sean-k-mooney | alex_xu: we generate teh metadtat here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/config.py#L2860 | |
| 14:45:05 | alex_xu | sean-k-mooney: I'm thinking we have uuid for flavor...? | |
| 14:45:16 | mriedem | alex_xu: and this is because the move claim for pmems is moving to the driver, right? | |
| 14:45:19 | dansmith | alex_xu: okay I don't understand, since the flavor id won't tell you size or type or anything, but I guess I'll understand when I read | |
| 14:45:39 | sean-k-mooney | alex_xu: yes we do but we also have other info | |
| 14:45:41 | Tianhao_Hu | mriedem: Thank you for your advice and I will get someone to work with me trying to find out why the directory is left. | |
| 14:45:43 | mriedem | alex_xu: flavor.flavorid is the user-facing thing | |
| 14:46:03 | mriedem | flavorid is not necessarily a uuid, | |
| 14:46:12 | mriedem | but it must be unique | |
| 14:46:16 | sean-k-mooney | actully we might not have the uuid unesll we put it in the namve field | |
| 14:46:20 | alex_xu | dansmith: flavor id will tell you it is src or dest. the dest is new flavor, the src is old flavor | |
| 14:46:23 | luyao | dansmith: flavor id is used to mark a namespace is assigned to which instance with which flavor | |
| 14:46:34 | dansmith | luyao: I think that's a bad plan | |
| 14:46:39 | sean-k-mooney | alex_xu: you cal always add whatever you need to https://github.com/openstack/nova/blob/master/nova/virt/libvirt/config.py#L2902-L2930 | |