Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-12
18:37:41 ildikov melwitt: is your question on shelve offload about the case when we shelve an instance with volume attached?
18:38:17 dansmith cburgess: this is the magic that causes object operations to go via some other service (i.e. conductor): https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L56
18:38:27 dansmith cburgess: without that, they always default to direct
18:38:38 cburgess Where direct = DB?
18:38:44 dansmith cburgess: yes
18:38:55 cburgess dansmith I'm going to go cry in the corner now. Thank you.
18:39:06 dansmith cburgess: uh, why? you wanted via conductor?
18:39:39 cburgess dansmith Well conductor would be better then the DB yes. Though they shouldn't be loading the objects directly like that. They should be using the API.
18:39:42 melwitt ildikov: yes, I think so. it's not really related to that patch but something I noticed during review
18:40:22 dansmith cburgess: why would conductor be better? you'd have to setup all the rpc stuff to do that thing
18:40:28 cburgess dansmith As in this isn't code in nova some place. This is something entirely external to nova that wants some data and its choosing to do so loading the objects directly.
18:40:33 dansmith cburgess: but agree, if you can get it efficiently from the api that's better
18:40:34 ildikov melwitt: this is how it looks like for attaching a volume to a shelved offloaded instance: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3720
18:41:17 cburgess dansmith My real issue is they should not be loading the objects at all. That needs to be via the API. Clients of the cloud don't get to load internal object code and muck about in the DB. They should talk to the API.
18:41:18 dansmith cburgess: ack, you're crying because someone who is paying you money is doing a scary thing and you have to figure out how to fix that and keep getting paid, is that right?
18:41:26 ildikov melwitt: we call attachment_create as part of the check_attach_and_reserve_volume call and in addition we call attachment_complete in _attach_volume_shelved_offloaded to put the volume to 'in-use'
18:41:29 dansmith cburgess: ack, yes
18:41:31 cburgess dansmith Something like that.
18:42:05 dansmith cburgess: just wanted to make sure you weren't complaining that my objects don't default to rpc or something :P
18:42:09 ildikov melwitt: it's honestly a workaround for now as the volume would be perfectly fine in 'reserved' state, but that's not how the old flow worked, so changing that needs an API microversion
18:42:34 cburgess dansmith Nope I love you objects, and the conductor.
18:42:42 dansmith hah
18:43:03 ildikov melwitt: when we do the offload the part when we call attachment_create is correct as attachment_delete detaches the volume completely
18:43:23 ildikov melwitt: what I'm unsure at the moment is whether we call attachment_complete in that case as well
18:43:39 melwitt ildikov: what I mean is, when we go to shelve an instance, we detach the volume. but with that recent change it's not only deleting the attachment, it creates a new attachment. which I didn't think we would want while an instance is shelved-offloaded
18:44:15 melwitt oh really? so we want to keep an attachment (create a new one) while shelve offloaded?
18:44:38 mriedem melwitt: yes,
18:44:42 mriedem that keeps the volume reserved
18:44:45 ildikov melwitt: but with shelve the old flow only removes the connection on the host but the volume is still appearing to be attached to that instance on the Cinder side
18:44:47 mriedem while the instance is shelved
18:45:08 melwitt okay, good then. sorry for the noise
18:45:12 mriedem the empty vol attachment keeps the volume reserved but that vol attachment isn't actually connected to any host
18:45:16 ildikov melwitt: so when you call attachment_delete in terminate_connections that moves to volume back to 'available'
18:45:33 mriedem unless there is another attachment on the volume already
18:45:34 ildikov mriedem: do we call attachment_complete in every bit of shelve, where we need to?
18:45:42 mriedem once the attachment count is 0, then the volume is 'available' again
18:46:10 mriedem ildikov: you mean unshelve?
18:46:28 mriedem ildikov: yes via https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4783
18:46:34 mriedem same as normal server create
18:46:53 mriedem happens down in the DriverVolumeBlockDevice.attach magic cauldron
18:47:15 mriedem magic cauldron of attachitude
18:47:24 mriedem henceforth shall it be known
18:47:35 ildikov mriedem: I meant whether we have actions as shelve and attaching a volume to a shelved instance and which call inside the Nova code covers these
18:47:56 ildikov mriedem: as we want the volume to be 'in-use' with the new flow as well so we added the additional attachment_complete call
18:48:05 mriedem attaching to a shelved offloaded instance happens in the api and we complete the attachment there too as you pointed out
18:48:37 mriedem remember that we can complete an empty attachment,
18:48:41 mriedem and update a completed attachment
18:48:48 mriedem weird as that may be
18:48:53 mriedem that's why unshelve works
18:49:07 mriedem *that's why unshelve works for the case that the volume is attached while the instance is shelved offloaded
18:51:49 ildikov mriedem: I understand that part
18:52:08 ildikov mriedem: I was just wondering whether we do all the steps when we shelve offload an instance wit volume attached
18:52:13 ildikov as I guess we can do that too
18:52:46 ildikov which might be the same set of calls under the hood, just wanted to double check after melwitt's confusion :)
18:54:33 mriedem yes, _prep_block_device happens it
18:54:36 mriedem on unshelve
18:54:40 mriedem *handles
18:56:24 ildikov ok, cool
18:56:50 ildikov thanks
19:04:05 mriedem sdague: where is the code for your api-ref burndown chart
19:04:06 mriedem ?
19:04:16 mriedem https://github.com/sdague/rst-burndown ?
19:05:17 cdent mriedem: gibi's sort of fork is at https://github.com/gibizer/nova-versioned-notification-transformation-burndown
19:05:24 mriedem yup, just found that too
19:05:27 sdague mriedem: yeh
19:05:41 mriedem mrhillsman is looking for something similar for sdk gap burndown
19:06:07 cdent I've got some totally 100% non-fancy infrastructure in place to run that stuff for gibi, if mrhillsman wants to get in on that: http://burndown.peermore.com/nova-notification/
19:06:19 cdent it runs a cron job to update itself
19:06:43 mriedem i'll pass that on thanks
19:06:47 cdent but presumably if this is for openlap, they've already got such stuff
19:06:56 cdent openlap, hmmm
19:07:02 cdent if that's not already started, probably ought to be
19:07:31 mriedem yeah i'm assuming he can host it himself
19:08:51 mriedem er yeah http://openlabtesting.org/
19:09:22 cdent even comes with uml
19:30:27 mriedem dansmith: want to check out the first 2 changes in the alternate hosts series here? https://review.openstack.org/#/c/516707/ - just needs final +2
19:30:48 dansmith okay in a bit
19:39:22 mriedem for anyone that cares, the instance action paging series is also ready to go i think https://review.openstack.org/#/q/topic:bp/pagination-add-changes-since-for-instance-action-list+status:open
19:39:50 mriedem i've started moving fully complete series up to the top of https://etherpad.openstack.org/p/nova-queens-blueprint-status
20:15:49 openstackgerrit Matt Riedemann proposed openstack/nova master: [WIP] POC to use neutron port_list when filtering instance by ip https://review.openstack.org/525505
20:18:43 openstackgerrit Michael Still proposed openstack/nova master: Convert ext filesystem resizes to privsep. https://review.openstack.org/517516
20:18:44 openstackgerrit Michael Still proposed openstack/nova master: Move flushing block devices to privsep. https://review.openstack.org/519010
20:18:44 openstackgerrit Michael Still proposed openstack/nova master: Start moving users of parted to privsep. https://review.openstack.org/519011
20:18:45 openstackgerrit Michael Still proposed openstack/nova master: Move remaining uses of parted to privsep. https://review.openstack.org/519483
20:18:45 openstackgerrit Michael Still proposed openstack/nova master: Convert users of tune2fs to privsep. https://review.openstack.org/519484
20:18:46 openstackgerrit Michael Still proposed openstack/nova master: Move makefs to privsep https://review.openstack.org/527510
20:51:45 mnaser throwing this up for discussion - how would folks feel if image caching used the imagebackend instead of just it's own cache
20:52:15 mnaser it would be beneficial for scenarios when you have a qcow2 image stored, with ceph backend for example, avoiding the import and convert on every iteration
20:53:01 mnaser another interesting scenario is multiple cells with independent ceph clusters (thereby, image is not directly accessible to do a cow new image)
20:53:55 mnaser (we're going to use cells v2 to add another az in an entirely different physical facility with a 10g link between them, fun times ahead :>)
20:56:26 melwitt mnaser: if you're using ceph and its native qcow2 format, you will be using glance images of type RAW and will get fast-clone and no image conversions, right?
20:56:48 melwitt that is, what do you mean by 'import and convert'?
20:56:55 mnaser melwitt: ceph with images stored in raw will get fast clone
20:57:33 mnaser melwitt: so - current scenario - images stored in raw in `images` pool in ceph, vm disks stored in `vms` pool in ceph. when nova boots a new vm, it can access `images` directly so it does a fast-clone and you get a disk very quickly
20:58:12 mnaser melwitt: if the image format is qcow2, ceph wont be able to do fast clone. nova will pull the image down locally, convert it to raw, and upload/import it back to ceph, then boot the machine
20:58:21 melwitt right
20:59:21 mnaser melwitt: now in an env where you have 2 cells, each one has a ceph cluster of its own, cell A which uses the same ceph cluster as glance can do fast clone. cell B will check the location and realize it has no way to reach that ceph cluster, so it will download the raw image locally and import/upload it to ceph
20:59:56 mnaser and for every new VM, it will do this (obviously leveraging local image cache), but that can be a lot of traffic before an image gets cached
21:00:39 mnaser my two ideas to work this around is... a) use the multiple locations feature of glance to mirror the images and add an rbd location in both clusters (nova loops over all locations till it finds one, then falls back to download)
21:00:46 melwitt from the ceph docs "Important Ceph doesn’t support QCOW2 for hosting a virtual machine disk. Thus if you want to boot virtual machines in Ceph (ephemeral backend or boot from volume), the Glance image format must be RAW." http://docs.ceph.com/docs/master/rbd/rbd-openstack/

Earlier   Later