Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-04
15:57:29 Cardoe yes
15:57:35 jmccarthy jgwentworth: But there must be logic to know when it's safe to remove that dir on the source host ?
15:57:38 jgwentworth jmccarthy: did you see the patch mriedem linked though? it's bypassing all of that if volume-backed https://review.openstack.org/#/c/516395/2/nova/virt/libvirt/driver.py because it was trying to fix the case where the console.log was being deleted
15:57:41 mriedem i have no idea if libvirt/hyperv/vmware support that
15:57:59 jmccarthy 1728603 ? Let me look
15:58:18 Cardoe mriedem: I've been told that I do not believe that they do.
15:58:18 jgwentworth no, https://review.openstack.org/#/c/516395/2/nova/virt/libvirt/driver.py
15:58:31 Cardoe s/I/rax OpenStack folks/
15:59:36 jmccarthy jswentworth: I think the driver.py that I have now looks like that one ?
16:00:01 jgwentworth jmccarthy: so because you are volume-backed, it's not removing anything at all. I think it needs to be adjusted to only delete disk.info if volume-backed, or something like that
16:00:14 mriedem jmccarthy: because you're on queens and that change was made in queens
16:00:31 mriedem and backported to pike and ocata because it was fixing a regression which was also backported to pike and ocata :)
16:00:31 jmccarthy jgwentworth: Ok so it's not just me - yes some logic is missing ?
16:00:35 mriedem it's bugs all the way down
16:00:52 jmccarthy jgwentworth: Leaving the dir there causes a problem for live migration too
16:00:57 jgwentworth yeah, there is definitely a bug. we were just trying to find what it was
16:01:07 mriedem jmccarthy: the dir or just the disk.info file itself?
16:01:13 jmccarthy jgwentworth: the dir
16:01:18 jmccarthy mriedman: the dir
16:01:30 jmccarthy Sorry I'm eventually getting there lol thanks for the help all ! :)
16:01:35 jmccarthy One sec
16:01:35 jgwentworth so what about the console.log then? where is it supposed to go
16:01:49 openstackgerrit Eric Fried proposed openstack/nova master: placement: Object changes for granular https://review.openstack.org/564351
16:01:50 openstackgerrit Eric Fried proposed openstack/nova master: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
16:02:03 jgwentworth oh, wait, supposed to remove the dir from the source
16:02:27 mriedem jgwentworth: but if you're on shared local storage, deleting the dir on the source will also remove it from the dest
16:02:29 jmccarthy mriedman: Actually I have it at the end of the bug - https://bugs.launchpad.net/nova/+bug/1769131 about 'already exists, it is expected not to exist.'
16:02:29 openstack Launchpad bug 1769131 in OpenStack Compute (nova) "After cold-migration of a volume-backed instance, disk.info file leftover on source host" [Undecided,Triaged]
16:02:48 jgwentworth mriedem: yeah, right. okay.
16:02:59 mriedem jmccarthy: yeah DestinationDiskExists only happens if compute says you're not on shared storage,
16:03:09 mriedem but since you're using image_type=rbd i'd think your computes would be using shared storage
16:03:12 jgwentworth yeah so it sounds like just need to delete disk.info, the presence of the dir should be okay right?
16:03:24 jmccarthy Should it ?
16:03:29 jmccarthy I only have 2 computes
16:03:39 jmccarthy if I try to live migrate back I get the complaint about the dir being there
16:03:44 mriedem jgwentworth: i don't think so https://github.com/openstack/nova/blob/4b0d0ea9f18139d58103a520a6a4e9119e19a4de/nova/virt/libvirt/driver.py#L7457
16:04:03 jgwentworth oh :\
16:04:06 mriedem jmccarthy: i guess you can use image type rbd for local storage and not having it be shared, but that seems weird
16:04:07 jmccarthy 1) cold migrate - it works, but disk.info is left on source
16:04:16 mriedem apparently people do it, but i'm not sure why you'd be using rbd and not sharing it
16:04:17 jmccarthy 2) try to live back fails due to dir :/
16:04:19 jgwentworth then how is this supposed to work with shared storage
16:04:48 jmccarthy mriedman: Not sure I follow, I'm using cinder for storage, not local ?
16:05:14 mriedem the console.log is stored on local disk for the compute host
16:05:15 mriedem not in cinder
16:05:16 jmccarthy Well after the live fails then it does: Cleaning up deleted instances with incomplete migration
16:05:16 jmccarthy _cleanup_incomplete_migrations
16:05:23 jmccarthy and the next attempt to live migrate works
16:05:44 jgwentworth okay yeah I see now, you are using rbd but not shared storage ... only does DestinationDiskExists if not shared
16:06:10 mriedem so is it just me, or does https://review.openstack.org/#/c/437356/3/nova/virt/libvirt/driver.py@1136 seem like kind of a hack?
16:06:13 jmccarthy mreidman: I'm confused, I do have a console log file there, but I am getting volumes also ..
16:06:21 Cardoe mriedem: what would you like to see happen with the rescue stuff? We can try and submit a patch to allow it for xenapi?
16:06:24 mriedem "self.image_backend.image for some backends recreates instance directory and image disk.info - remove it here if exists"
16:06:43 jgwentworth mriedem: yeah, it does. when I read it I was like, "why is imagebackend sometimes recreating the dir"
16:06:46 mriedem Cardoe: idk yet, that's why i posted to the ML
16:06:57 jmccarthy s/getting/using
16:07:05 mriedem those details are in https://bugs.launchpad.net/nova/+bug/1666831
16:07:05 openstack Launchpad bug 1666831 in OpenStack Compute (nova) ocata "Nova recreates instance directory after migration/resize" [Low,Fix committed] - Assigned to Lee Yarwood (lyarwood)
16:08:14 mriedem apparently this recreates the disk.info on the source host https://github.com/openstack/nova/blob/4b0d0ea9f18139d58103a520a6a4e9119e19a4de/nova/virt/libvirt/driver.py#L1152
16:08:31 mriedem and then https://github.com/openstack/nova/blob/4b0d0ea9f18139d58103a520a6a4e9119e19a4de/nova/virt/libvirt/driver.py#L1170 was added to delete it
16:09:29 Cardoe mriedem: I've been told that the reason why we didn't upstream the change is that there was no easy way to allow it only for xenapi at the time.
16:09:33 jmccarthy For me this happens, and it's ok
16:09:34 jmccarthy CMD "rm -rf /var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294_resize" returned: 0
16:09:34 jmccarthy except then
16:09:49 jmccarthy Lock "/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294/disk.info" acquired
16:10:01 jmccarthy and Lock "/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294/disk.info" released by "nova.virt.libvirt.imagebackend.write_to_disk_info_file"
16:10:10 jmccarthy recreates it from what I can tell ?
16:10:22 jgwentworth yeah, I think so
16:11:12 jgwentworth Cardoe: is rebooting a rescued instance only safe or something that makes sense for xenapi?
16:11:17 mriedem right, that's what bug 1666831 is saying,
16:11:17 openstack bug 1666831 in OpenStack Compute (nova) ocata "Nova recreates instance directory after migration/resize" [Low,Fix committed] https://launchpad.net/bugs/1666831 - Assigned to Lee Yarwood (lyarwood)
16:11:18 mriedem and was trying to fix
16:11:38 mriedem to summarize, the libvirt driver and the imagebackend are totally inbred and shitty
16:11:43 mriedem mdbooth: ^?
16:11:57 mriedem the driver needs the image backend for rbd to remove snapshots,
16:12:10 mriedem but for qcow2 imagebackend it recreates the disk.info,
16:12:24 mriedem so then the libvirt driver deletes the directory that imagebackend.by_name() for qcow2 created
16:12:50 mriedem the control flow logic in the driver _cleanup_resize method is totally tightly coupled based on the image backend being used
16:13:05 jgwentworth yeah ... sucks
16:13:40 Cardoe jgwentworth: No. It'd make sense for others to get it. How its been explained to me is that when someone screws up their grub or whatever bootloader and they can't boot their instance it can go into a rescue mode where their disk is attached to a read-only Linux instance where they can fix it themselves. Then when its rebooted it will go back to their original configuration, but hopefully this time with a fixed boot loader.
16:14:47 mriedem Cardoe: so this isn't a case where the user literally calls the rescue API on the instance and then reboots it?
16:15:03 mriedem it's server create -> fails -> xen rescue mode shenanigans -> user reboots the instance
16:15:34 Cardoe create or software update
16:15:50 mriedem "software update" being something that happens within the xen guest?
16:15:53 mriedem that's not a nova term
16:16:03 mriedem anyway, none of that sounds like it puts the instance vm_state into RESCUED
16:16:12 mriedem which is what the compute manager code is checking
16:16:32 Cardoe When the instance fails to boot, the rescue API can be called to put it into rescued.
16:16:35 jgwentworth looks like cfriesen might have highlighted why reboot is rejected in his reply to the ML, reboot could make it impossible to unrescue the instance?
16:17:06 jgwentworth "loss of original instance state"
16:17:21 mriedem jgwentworth: jmccarthy: so i have an idea about how to handle this disk.info issue,
16:17:32 mriedem let me wip something up here and we can take a gander
16:17:32 jmccarthy mriedman: Ok cool :)
16:18:03 jgwentworth mriedem: sounds good
16:19:36 mriedem Cardoe: if you're subscribed to the openstack-dev ML, it would be cool if you could reply with details on the xenapi background in that thread
16:19:42 mriedem i've got some code to break
16:20:46 jmccarthy mriedman: I did try running the tox tests in this area, either I ran the wrong tests, or this setup isn't quite covered
16:20:47 Cardoe I'm not. I certainly can as I get more info. sorry its a bad game of telephone with me. If you guys wanted to talk assembly and hypercalls I'm you're guy. APIs for orchestrating VMs not so much.
16:25:32 mriedem i haven't wanted to talk assembly in 19 years
16:25:38 jmccarthy Is it a misconfiguration on my part to have images_type = rbd if I'm not using ceph ?

Earlier   Later