Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-04
16:01:49 openstackgerrit Eric Fried proposed openstack/nova master: placement: Object changes for granular https://review.openstack.org/564351
16:01:50 openstackgerrit Eric Fried proposed openstack/nova master: placement: Granular GET /allocation_candidates https://review.openstack.org/517757
16:02:03 jgwentworth oh, wait, supposed to remove the dir from the source
16:02:27 mriedem jgwentworth: but if you're on shared local storage, deleting the dir on the source will also remove it from the dest
16:02:29 openstack Launchpad bug 1769131 in OpenStack Compute (nova) "After cold-migration of a volume-backed instance, disk.info file leftover on source host" [Undecided,Triaged]
16:02:29 jmccarthy mriedman: Actually I have it at the end of the bug - https://bugs.launchpad.net/nova/+bug/1769131 about 'already exists, it is expected not to exist.'
16:02:48 jgwentworth mriedem: yeah, right. okay.
16:02:59 mriedem jmccarthy: yeah DestinationDiskExists only happens if compute says you're not on shared storage,
16:03:09 mriedem but since you're using image_type=rbd i'd think your computes would be using shared storage
16:03:12 jgwentworth yeah so it sounds like just need to delete disk.info, the presence of the dir should be okay right?
16:03:24 jmccarthy Should it ?
16:03:29 jmccarthy I only have 2 computes
16:03:39 jmccarthy if I try to live migrate back I get the complaint about the dir being there
16:03:44 mriedem jgwentworth: i don't think so https://github.com/openstack/nova/blob/4b0d0ea9f18139d58103a520a6a4e9119e19a4de/nova/virt/libvirt/driver.py#L7457
16:04:03 jgwentworth oh :\
16:04:06 mriedem jmccarthy: i guess you can use image type rbd for local storage and not having it be shared, but that seems weird
16:04:07 jmccarthy 1) cold migrate - it works, but disk.info is left on source
16:04:16 mriedem apparently people do it, but i'm not sure why you'd be using rbd and not sharing it
16:04:17 jmccarthy 2) try to live back fails due to dir :/
16:04:19 jgwentworth then how is this supposed to work with shared storage
16:04:48 jmccarthy mriedman: Not sure I follow, I'm using cinder for storage, not local ?
16:05:14 mriedem the console.log is stored on local disk for the compute host
16:05:15 mriedem not in cinder
16:05:16 jmccarthy _cleanup_incomplete_migrations
16:05:16 jmccarthy Well after the live fails then it does: Cleaning up deleted instances with incomplete migration
16:05:23 jmccarthy and the next attempt to live migrate works
16:05:44 jgwentworth okay yeah I see now, you are using rbd but not shared storage ... only does DestinationDiskExists if not shared
16:06:10 mriedem so is it just me, or does https://review.openstack.org/#/c/437356/3/nova/virt/libvirt/driver.py@1136 seem like kind of a hack?
16:06:13 jmccarthy mreidman: I'm confused, I do have a console log file there, but I am getting volumes also ..
16:06:21 Cardoe mriedem: what would you like to see happen with the rescue stuff? We can try and submit a patch to allow it for xenapi?
16:06:24 mriedem "self.image_backend.image for some backends recreates instance directory and image disk.info - remove it here if exists"
16:06:43 jgwentworth mriedem: yeah, it does. when I read it I was like, "why is imagebackend sometimes recreating the dir"
16:06:46 mriedem Cardoe: idk yet, that's why i posted to the ML
16:06:57 jmccarthy s/getting/using
16:07:05 openstack Launchpad bug 1666831 in OpenStack Compute (nova) ocata "Nova recreates instance directory after migration/resize" [Low,Fix committed] - Assigned to Lee Yarwood (lyarwood)
16:07:05 mriedem those details are in https://bugs.launchpad.net/nova/+bug/1666831
16:08:14 mriedem apparently this recreates the disk.info on the source host https://github.com/openstack/nova/blob/4b0d0ea9f18139d58103a520a6a4e9119e19a4de/nova/virt/libvirt/driver.py#L1152
16:08:31 mriedem and then https://github.com/openstack/nova/blob/4b0d0ea9f18139d58103a520a6a4e9119e19a4de/nova/virt/libvirt/driver.py#L1170 was added to delete it
16:09:29 Cardoe mriedem: I've been told that the reason why we didn't upstream the change is that there was no easy way to allow it only for xenapi at the time.
16:09:33 jmccarthy For me this happens, and it's ok
16:09:34 jmccarthy except then
16:09:34 jmccarthy CMD "rm -rf /var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294_resize" returned: 0
16:09:49 jmccarthy Lock "/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294/disk.info" acquired
16:10:01 jmccarthy and Lock "/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294/disk.info" released by "nova.virt.libvirt.imagebackend.write_to_disk_info_file"
16:10:10 jmccarthy recreates it from what I can tell ?
16:10:22 jgwentworth yeah, I think so
16:11:12 jgwentworth Cardoe: is rebooting a rescued instance only safe or something that makes sense for xenapi?
16:11:17 openstack bug 1666831 in OpenStack Compute (nova) ocata "Nova recreates instance directory after migration/resize" [Low,Fix committed] https://launchpad.net/bugs/1666831 - Assigned to Lee Yarwood (lyarwood)
16:11:17 mriedem right, that's what bug 1666831 is saying,
16:11:18 mriedem and was trying to fix
16:11:38 mriedem to summarize, the libvirt driver and the imagebackend are totally inbred and shitty
16:11:43 mriedem mdbooth: ^?
16:11:57 mriedem the driver needs the image backend for rbd to remove snapshots,
16:12:10 mriedem but for qcow2 imagebackend it recreates the disk.info,
16:12:24 mriedem so then the libvirt driver deletes the directory that imagebackend.by_name() for qcow2 created
16:12:50 mriedem the control flow logic in the driver _cleanup_resize method is totally tightly coupled based on the image backend being used
16:13:05 jgwentworth yeah ... sucks
16:13:40 Cardoe jgwentworth: No. It'd make sense for others to get it. How its been explained to me is that when someone screws up their grub or whatever bootloader and they can't boot their instance it can go into a rescue mode where their disk is attached to a read-only Linux instance where they can fix it themselves. Then when its rebooted it will go back to their original configuration, but hopefully this time with a fixed boot loader.
16:14:47 mriedem Cardoe: so this isn't a case where the user literally calls the rescue API on the instance and then reboots it?
16:15:03 mriedem it's server create -> fails -> xen rescue mode shenanigans -> user reboots the instance
16:15:34 Cardoe create or software update
16:15:50 mriedem "software update" being something that happens within the xen guest?
16:15:53 mriedem that's not a nova term
16:16:03 mriedem anyway, none of that sounds like it puts the instance vm_state into RESCUED
16:16:12 mriedem which is what the compute manager code is checking
16:16:32 Cardoe When the instance fails to boot, the rescue API can be called to put it into rescued.
16:16:35 jgwentworth looks like cfriesen might have highlighted why reboot is rejected in his reply to the ML, reboot could make it impossible to unrescue the instance?
16:17:06 jgwentworth "loss of original instance state"
16:17:21 mriedem jgwentworth: jmccarthy: so i have an idea about how to handle this disk.info issue,
16:17:32 jmccarthy mriedman: Ok cool :)
16:17:32 mriedem let me wip something up here and we can take a gander
16:18:03 jgwentworth mriedem: sounds good
16:19:36 mriedem Cardoe: if you're subscribed to the openstack-dev ML, it would be cool if you could reply with details on the xenapi background in that thread
16:19:42 mriedem i've got some code to break
16:20:46 jmccarthy mriedman: I did try running the tox tests in this area, either I ran the wrong tests, or this setup isn't quite covered
16:20:47 Cardoe I'm not. I certainly can as I get more info. sorry its a bad game of telephone with me. If you guys wanted to talk assembly and hypercalls I'm you're guy. APIs for orchestrating VMs not so much.
16:25:32 mriedem i haven't wanted to talk assembly in 19 years
16:25:38 jmccarthy Is it a misconfiguration on my part to have images_type = rbd if I'm not using ceph ?
16:26:11 mriedem jmccarthy: well what local ephemeral storage on your compute hosts are you using for non-volume-backed instances?
16:26:42 jmccarthy I don't have anything setup specifically, local disk on computes I guess ?
16:26:52 jgwentworth I would think so, rbd == ceph
16:26:55 superdan mriedem: do we not store network_request(s) today?
16:27:02 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.images_type
16:27:08 mriedem superdan: not in the request spec no
16:27:09 mriedem nor bdms
16:27:14 superdan mriedem: I mean at all
16:27:18 mriedem correct
16:27:36 superdan mriedem: don't we need that on rebuild/evac? or do we get it from neutron or something?
16:27:51 mriedem we have the network info cache / neutron
16:27:54 superdan surely not just the info_cache at that point...?
16:27:57 mriedem you can't request new ports on rebuild/evac
16:27:59 superdan hrm
16:28:20 superdan mriedem: so, storing it in the reqspec.. that may potentially include port uuids that you booted with right?
16:28:30 mriedem the original network request list woudn't have things attached to the instance after it was created either
16:28:39 mriedem superdan: port id or network id yeah
16:28:39 superdan mriedem: what happens if/when I attach a new port and detach the old one.. now the reqspec has a stale from-two-years-ago port uuid in it right?
16:29:07 mriedem superdan: yeah, same could be said about the security groups right?
16:29:20 superdan yes, although that seems less problematic to me,
16:29:33 mriedem the spec says they aren't going to persist this anyway, just include it in the request spec is a middleman to get from the api to the scheduler
16:29:34 superdan probably because port_uuids are somewhat ephemeral

Earlier   Later