Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-04
16:10:01 jmccarthy and Lock "/var/lib/nova/instances/371e669b-0f15-49f2-9a84-bd1e89f34294/disk.info" released by "nova.virt.libvirt.imagebackend.write_to_disk_info_file"
16:10:10 jmccarthy recreates it from what I can tell ?
16:10:22 jgwentworth yeah, I think so
16:11:12 jgwentworth Cardoe: is rebooting a rescued instance only safe or something that makes sense for xenapi?
16:11:17 mriedem right, that's what bug 1666831 is saying,
16:11:17 openstack bug 1666831 in OpenStack Compute (nova) ocata "Nova recreates instance directory after migration/resize" [Low,Fix committed] https://launchpad.net/bugs/1666831 - Assigned to Lee Yarwood (lyarwood)
16:11:18 mriedem and was trying to fix
16:11:38 mriedem to summarize, the libvirt driver and the imagebackend are totally inbred and shitty
16:11:43 mriedem mdbooth: ^?
16:11:57 mriedem the driver needs the image backend for rbd to remove snapshots,
16:12:10 mriedem but for qcow2 imagebackend it recreates the disk.info,
16:12:24 mriedem so then the libvirt driver deletes the directory that imagebackend.by_name() for qcow2 created
16:12:50 mriedem the control flow logic in the driver _cleanup_resize method is totally tightly coupled based on the image backend being used
16:13:05 jgwentworth yeah ... sucks
16:13:40 Cardoe jgwentworth: No. It'd make sense for others to get it. How its been explained to me is that when someone screws up their grub or whatever bootloader and they can't boot their instance it can go into a rescue mode where their disk is attached to a read-only Linux instance where they can fix it themselves. Then when its rebooted it will go back to their original configuration, but hopefully this time with a fixed boot loader.
16:14:47 mriedem Cardoe: so this isn't a case where the user literally calls the rescue API on the instance and then reboots it?
16:15:03 mriedem it's server create -> fails -> xen rescue mode shenanigans -> user reboots the instance
16:15:34 Cardoe create or software update
16:15:50 mriedem "software update" being something that happens within the xen guest?
16:15:53 mriedem that's not a nova term
16:16:03 mriedem anyway, none of that sounds like it puts the instance vm_state into RESCUED
16:16:12 mriedem which is what the compute manager code is checking
16:16:32 Cardoe When the instance fails to boot, the rescue API can be called to put it into rescued.
16:16:35 jgwentworth looks like cfriesen might have highlighted why reboot is rejected in his reply to the ML, reboot could make it impossible to unrescue the instance?
16:17:06 jgwentworth "loss of original instance state"
16:17:21 mriedem jgwentworth: jmccarthy: so i have an idea about how to handle this disk.info issue,
16:17:32 mriedem let me wip something up here and we can take a gander
16:17:32 jmccarthy mriedman: Ok cool :)
16:18:03 jgwentworth mriedem: sounds good
16:19:36 mriedem Cardoe: if you're subscribed to the openstack-dev ML, it would be cool if you could reply with details on the xenapi background in that thread
16:19:42 mriedem i've got some code to break
16:20:46 jmccarthy mriedman: I did try running the tox tests in this area, either I ran the wrong tests, or this setup isn't quite covered
16:20:47 Cardoe I'm not. I certainly can as I get more info. sorry its a bad game of telephone with me. If you guys wanted to talk assembly and hypercalls I'm you're guy. APIs for orchestrating VMs not so much.
16:25:32 mriedem i haven't wanted to talk assembly in 19 years
16:25:38 jmccarthy Is it a misconfiguration on my part to have images_type = rbd if I'm not using ceph ?
16:26:11 mriedem jmccarthy: well what local ephemeral storage on your compute hosts are you using for non-volume-backed instances?
16:26:42 jmccarthy I don't have anything setup specifically, local disk on computes I guess ?
16:26:52 jgwentworth I would think so, rbd == ceph
16:26:55 superdan mriedem: do we not store network_request(s) today?
16:27:02 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.images_type
16:27:08 mriedem superdan: not in the request spec no
16:27:09 mriedem nor bdms
16:27:14 superdan mriedem: I mean at all
16:27:18 mriedem correct
16:27:36 superdan mriedem: don't we need that on rebuild/evac? or do we get it from neutron or something?
16:27:51 mriedem we have the network info cache / neutron
16:27:54 superdan surely not just the info_cache at that point...?
16:27:57 mriedem you can't request new ports on rebuild/evac
16:27:59 superdan hrm
16:28:20 superdan mriedem: so, storing it in the reqspec.. that may potentially include port uuids that you booted with right?
16:28:30 mriedem the original network request list woudn't have things attached to the instance after it was created either
16:28:39 superdan mriedem: what happens if/when I attach a new port and detach the old one.. now the reqspec has a stale from-two-years-ago port uuid in it right?
16:28:39 mriedem superdan: port id or network id yeah
16:29:07 mriedem superdan: yeah, same could be said about the security groups right?
16:29:20 superdan yes, although that seems less problematic to me,
16:29:33 mriedem the spec says they aren't going to persist this anyway, just include it in the request spec is a middleman to get from the api to the scheduler
16:29:34 superdan probably because port_uuids are somewhat ephemeral
16:29:48 mriedem i'm fine with not persisting it
16:30:04 superdan mriedem: okay in general I think it's confusing to put things in DB-persisted objects that we don't store in the DB
16:30:18 mriedem but i was thinking for modeling, NetworkRequest object gains a RequestGroup field, and then RequestSpec gains a (non-persisted) NetworkRequestList field
16:30:20 superdan someone sets something there and calls save(), no error, so expects it's saved now, but isn't
16:30:45 superdan what is requestgroup?
16:30:49 mriedem ok, gibi has an alternative in the spec that it doesn't go into the request spec at all, and it's passed as a separate param to select_destinations()
16:31:02 mriedem it's in the placement lib - contains things about a granular request group
16:31:03 jmccarthy Ah sorry ! I was looking at a template, not the final rendered file
16:31:11 superdan oh, that right
16:31:19 mriedem so they'd have one of those per port
16:31:32 jmccarthy For my deployed nova.conf there is no entry there for images_type at all (so I guess default?)
16:31:35 superdan mriedem: well, a new rpc argument to the scheduler just for network stuff is also kinda odd, which I guess was your point
16:31:36 mriedem if it doesn't go in the request spec because we don't persist it, then it's a new param to select_destinations()
16:32:16 mriedem jgwentworth: so i don't think my idea will work..drats
16:32:56 mriedem jgwentworth: my idea was just to check https://github.com/openstack/nova/blob/4b0d0ea9f18139d58103a520a6a4e9119e19a4de/nova/virt/libvirt/imagebackend.py#L57 in driver._cleanup_resize and if True, do the snapshot removal stuff that is rbd-specific
16:33:01 jgwentworth *sad trombone*
16:33:14 mriedem but to get that value, we have to init the imagebackend object which is the thing that recreates the gd disk.info file
16:33:41 mriedem so, we could make a module-level dict of image types that support clone and use that,
16:33:57 mriedem or pass a flag to imagebackend init to tell it to not touch the filesystem
16:34:04 mriedem former seems easier
16:34:19 mriedem or hell, just: if CONF.libvirt.images_type == 'rbd': in _cleanup_resize
16:34:30 mriedem that's essentially what we'd be doing if we check SUPPORTS_CLONE
16:37:04 jgwentworth do you mean change the check for is_volume_backed to essentially "is_shared_storage"?
16:38:58 mriedem that entire block goes away
16:39:07 jmccarthy <back asap - afk>
16:39:13 mriedem i.e. basically undo this https://review.openstack.org/#/c/437356/3/nova/virt/libvirt/driver.py
16:39:17 mriedem and fix the logic
16:39:24 jgwentworth oh, I see, yeah
16:39:34 jgwentworth instead of using root_disk.exists()
16:39:50 mriedem "if os.path.exists(inst_base) and not root_disk.exists():" was added for qcow2/flat/ploop because those will recreate the instance dir and disk.info on init,
16:40:01 mriedem when all we needed the imagebackend for was the remove_snap call,
16:40:04 mriedem which is rbd-specific
16:40:07 jgwentworth right
16:40:12 jgwentworth gotcha
16:40:19 mriedem so if we just don't create the imagebackend object in the first place, we avoid the init et al
16:40:22 mriedem \o/
16:40:30 jgwentworth yeah, seriously
16:40:44 mriedem i of course will require jmccarthy to test the patch in his setup
16:41:00 jgwentworth oh, because you have to use it to do the remove_snap
16:41:27 mriedem yup and that's pointed out in https://bugs.launchpad.net/nova/+bug/1666831
16:41:27 openstack Launchpad bug 1666831 in OpenStack Compute (nova) ocata "Nova recreates instance directory after migration/resize" [Low,Fix committed] - Assigned to Lee Yarwood (lyarwood)
16:41:34 mriedem "root_disk is used to remove rdb snapshots, but during execution of self.image_backend.by_name() nova recreates instance directory."
16:41:50 jgwentworth so how can we remove_snap without the imagebackend object?

Earlier   Later