Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-04-03
12:19:38 sean-k-mooney if the storage backend supports that
12:37:41 opendevreview liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338
12:47:52 damnthem sean-k-mooney: vm works locally from {instance}/disk isn't it? Also domxml and qemu-img info doesnt show any backing chains for mirror/delta. And dev.rebase creates full copy of image (mirror https://qemu-project.gitlab.io/qemu/interop/live-block-operations.html#live-disk-synchronization-drive-mirror-and-blockdev-mirror) https://opendev.org/openstack/nova/src/commit/29de62bf3b3bf5eda8986bc94babf1c94d67bd4e/nova/virt/libvirt/d
12:48:53 opendevreview liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338
12:50:46 sean-k-mooney damnthem: the specific way we create snapshots is rather complicated and is done in a specic way for both legacy and technical reasons related to minimising the impact on the running guest
12:51:56 Uggla gibi, Hi I'm looking at https://review.opendev.org/c/openstack/nova/+/855664/ , so it needs to be backported down to train ?
12:53:59 sean-k-mooney damnthem: https://github.com/openstack/nova/blob/29de62bf3b3bf5eda8986bc94babf1c94d67bd4e/nova/virt/libvirt/driver.py#L3286-L3369 this is the relevent code
12:57:34 sean-k-mooney damnthem: https://github.com/openstack/nova/commit/46de2d1e2d0abd6fdcd4da13facaf3225c721f5e was the orgianl patch that add live-snapshots and it descibe the limitation around blockrebase and shallow copies
12:59:24 sean-k-mooney we later optimised this futher in https://github.com/openstack/nova/commit/caf5faf55670ab212868498e421bedc074fafd89 to reduce the time we need to freeze the fs
13:02:25 damnthem sean-k-mooney: Yeah i read that, and that's actually source of my confusion. Commit message says: "This process ultimately produces a CoW file, representing only the current delta between the root disk and backing file". But it's not in current state. There are 3 full copy at some point: instance disk, mirror and converted image from mirror.
13:08:48 sean-k-mooney i need to prepare for a meeting shortly so perhaps others can continue this converstation. without digining into the code my understanding of the process a a high level is we do somthign like this
13:09:26 sean-k-mooney first we create a delat disk by using the same backing as the vm image
13:10:01 sean-k-mooney this basically recreates teh state to the vm before it started runing for the first tiem
13:11:21 sean-k-mooney we then do a blockdevie rebase on the delta disk to update it with the changes that have hppened since the vm booted.
13:12:04 sean-k-mooney we then freeze the guest filesystem and abort the rebase job
13:12:09 sean-k-mooney instead of commiting it
13:12:20 sean-k-mooney then unfreeze the filesystem
13:12:44 sean-k-mooney at this point the vm is back to runing form the orginal instance disk
13:13:09 sean-k-mooney and we have a copy of the filesytem changes in the delta disk
13:14:19 sean-k-mooney we then update the ownwershp of the delta disk so that its owned by nova and revert the guest xml back ot what it was before the snapshot
13:14:45 sean-k-mooney then we flaten the delta disk into the final format for uploading
13:14:53 sean-k-mooney and delete the delta disk
13:15:02 sean-k-mooney finally we upload the image to glance.
13:15:56 sean-k-mooney by creating the delta disk as an overlay of the backing file durint the snapshot we only need enough storage for the changes since the guest booted
13:16:22 damnthem sean-k-mooney: thank you. I think i understood where is difference in my case and why it's works irrational for me.
13:16:24 sean-k-mooney while we are doing the format convertion we need storage for the vm + deta disk + flattened image
13:17:03 sean-k-mooney and after the snap shot we are just back to the orginal disk
13:33:55 damnthem sean-k-mooney: to be short - we disabled used_cow_images, so backing stores disabled .So this whole image/snapshot juggling looks pointless from outside. Thank you for helping me figure this out
13:40:22 sean-k-mooney damnthem: so your using raw images
13:40:45 sean-k-mooney this is still useful in this case for older release as it reduces the time the mirror action takes
14:39:45 opendevreview Alexey Stupnikov proposed openstack/nova master: Process unlimited exceptions raised by unplug_vifs https://review.opendev.org/c/openstack/nova/+/879350
14:41:48 auniyal Hi dansmith
14:41:55 dansmith hi
14:41:56 auniyal regarding this: https://review.opendev.org/c/openstack/nova-specs/+/878757
14:42:02 auniyal thanks for looking :)
14:42:11 auniyal as you said, now I too think the best way to update DB is with process of restarting instnace.
14:42:21 auniyal I was looking for the place where, we can updated DB, on instance shutoff.
14:42:33 auniyal I think it should be in compute/api as it has stop and start functionality.
14:42:50 auniyal I was also looking for list of operation nova might perform on instance shutdown or start, but could not find one.
14:43:02 auniyal Is this alright to add a decorator which perform all operations on instnace shutoff
14:43:15 auniyal something like
14:43:20 auniyal pass
14:43:20 auniyal def stop(instance):
14:43:20 auniyal @post_shutoff_actions
14:44:47 dansmith auniyal: compute/api is run on the caller, which means the api service would run that code for stop(), so no, I don't think that's best place
14:44:50 dansmith should be in manager
14:46:01 auniyal at this https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L3317
15:02:58 dansmith auniyal: perhaps, but it probably would be better on start, but more like the lower level spawn so that it catches reboot, start, etc
15:05:05 auniyal dansmith, ack,
15:06:07 auniyal I didn't get, why we should not run DB update at caller (i.e controller as nova-api serice I believe !!)
15:06:25 auniyal is it because this action is DB related
15:08:15 dansmith no, for several reasons:
15:08:46 dansmith It needs to call to cinder and the database so it may be slow/blocking and holding the API caller while you do that is not good
15:09:09 dansmith it also means that it would only work for an actual api stop call and not for other things like reboot, or crash recovery, or guest-initiated reboot, etc
15:11:17 auniyal ack, got it,
15:37:30 opendevreview Alexey Stupnikov proposed openstack/nova master: Process unlimited exceptions raised by unplug_vifs https://review.opendev.org/c/openstack/nova/+/879350
15:46:58 gibi Uggla: I we we need https://review.opendev.org/c/openstack/nova/+/855664/ in train yes
15:47:18 gibi Uggla: is there a complication with the backport?
15:47:42 Uggla gibi, all ports should be upstream ? Any downstream to do ?
15:49:11 Uggla gibi, no I just try to check the "scope".
15:57:39 gibi Uggla: as we have train open upstream still we expected to land the fix upstream.
15:58:15 gibi if there is high pressure to get the fix earlier downstream then you can propose the downstream backport before the upstream backport lands, but we still need the upstream backport too
15:58:34 gibi (we probably discuss this downstream :D)
15:59:14 Uggla ok sounds good.
15:59:54 gibi Uggla: thanks for picking that fix up
16:00:32 Uggla gibi, you are welcome.
16:39:30 gibi :)
#openstack-nova - 2023-04-04
02:09:15 opendevreview liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338
02:15:20 opendevreview liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338
04:45:37 opendevreview liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338
08:38:32 opendevreview Konrad Gube proposed openstack/nova-specs master: Re-propose using extend volume completion action for 2023.2 https://review.opendev.org/c/openstack/nova-specs/+/877233
08:50:40 kgube Hi bauzas, could you have a look at my spec? https://review.opendev.org/c/openstack/nova-specs/+/877233
08:50:45 kgube You had some issues with the implemtentation for Antelope that might be better to discuss at the spec level: https://review.opendev.org/c/openstack/nova/+/873560
08:50:58 bauzas kgube: ack, will try to do todaty
08:51:22 kgube thanks!
11:20:43 sahid lajoskatona: o/ if that sounds right for you I'm taking the lead to migrate nova on openstacksdk https://etherpad.opendev.org/p/python-neutronclient_deprecation
11:54:29 lajoskatona sahid: cool, thanks for checking it
12:23:30 opendevreview Alexey Stupnikov proposed openstack/nova stable/xena: Add debug log for scheduler weight calculation https://review.opendev.org/c/openstack/nova/+/879404
12:24:26 opendevreview Alexey Stupnikov proposed openstack/nova stable/wallaby: Add debug log for scheduler weight calculation https://review.opendev.org/c/openstack/nova/+/879405
13:33:08 dansmith bauzas: what regression do you think the long-wait change introduced?
13:33:38 bauzas dansmith: well, I don't really know but the test you modified was hit
13:34:03 dansmith uh, okay :)
13:36:34 dansmith bauzas: it's failing on create server, I only modified the attach volume line
13:36:41 dansmith so I think it's not likely related
13:36:44 bauzas ok
13:50:54 wangrong https://review.opendev.org/c/openstack/nova-specs/+/877291
13:50:54 wangrong dansmith: hello Dan, based on our previous agreement, we have prepared the relevant spec and would like you to review them. If you have any questions, please contact with us anytime. Thank you!
13:51:41 dansmith wangrong: that's my spec that I showed you as an example.. did you paste the wrong link?
13:53:30 wangrong dansmith: oh, sorry, my bad...
13:53:47 wangrong dansmith: https://review.opendev.org/c/openstack/nova-specs/+/879338
13:54:12 wangrong dansmith: I think this the one we posted
13:54:32 dansmith wangrong: cool, sean-k-mooney is the one that needs to do most of the review, but I'll try to take a first stab later today as well
13:54:49 dansmith bauzas: also cc ^ as discussed maybe you can provide some early feedback like we promised
13:54:58 bauzas cool
13:57:14 wangrong sure, thank you very much, so appreciate your help, dansmith, bauzas and sean-k-mooney .
14:02:45 bauzas as a reminder, due to European DST change one week before, the nova meeting will be in ~2 hours
14:03:13 bauzas (still the same UTC time, in case you're not in Europe)
15:11:19 TheJulia https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b1d/879215/3/check/ironic-grenade/b1d6cf4/controller/logs/screen-n-api.txt
15:11:19 TheJulia Are there any known issues on grenade from 2023.1 to master in regards to flavors? Specifically we're seeing a ~40% failure rate for grenade jobs on ironic, nova is specifically reporting the flavor baremetal is not found when it was added 5+ minutes earlier in the logs. Looks like post-create (200 response code), a lookup for it immediately fails with a 404.
15:17:12 TheJulia and the returned flavor does indicate it is public, btw.

Earlier   Later