| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-03 | |||
| 12:47:52 | damnthem | sean-k-mooney: vm works locally from {instance}/disk isn't it? Also domxml and qemu-img info doesnt show any backing chains for mirror/delta. And dev.rebase creates full copy of image (mirror https://qemu-project.gitlab.io/qemu/interop/live-block-operations.html#live-disk-synchronization-drive-mirror-and-blockdev-mirror) https://opendev.org/openstack/nova/src/commit/29de62bf3b3bf5eda8986bc94babf1c94d67bd4e/nova/virt/libvirt/d | |
| 12:48:53 | opendevreview | liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338 | |
| 12:50:46 | sean-k-mooney | damnthem: the specific way we create snapshots is rather complicated and is done in a specic way for both legacy and technical reasons related to minimising the impact on the running guest | |
| 12:51:56 | Uggla | gibi, Hi I'm looking at https://review.opendev.org/c/openstack/nova/+/855664/ , so it needs to be backported down to train ? | |
| 12:53:59 | sean-k-mooney | damnthem: https://github.com/openstack/nova/blob/29de62bf3b3bf5eda8986bc94babf1c94d67bd4e/nova/virt/libvirt/driver.py#L3286-L3369 this is the relevent code | |
| 12:57:34 | sean-k-mooney | damnthem: https://github.com/openstack/nova/commit/46de2d1e2d0abd6fdcd4da13facaf3225c721f5e was the orgianl patch that add live-snapshots and it descibe the limitation around blockrebase and shallow copies | |
| 12:59:24 | sean-k-mooney | we later optimised this futher in https://github.com/openstack/nova/commit/caf5faf55670ab212868498e421bedc074fafd89 to reduce the time we need to freeze the fs | |
| 13:02:25 | damnthem | sean-k-mooney: Yeah i read that, and that's actually source of my confusion. Commit message says: "This process ultimately produces a CoW file, representing only the current delta between the root disk and backing file". But it's not in current state. There are 3 full copy at some point: instance disk, mirror and converted image from mirror. | |
| 13:08:48 | sean-k-mooney | i need to prepare for a meeting shortly so perhaps others can continue this converstation. without digining into the code my understanding of the process a a high level is we do somthign like this | |
| 13:09:26 | sean-k-mooney | first we create a delat disk by using the same backing as the vm image | |
| 13:10:01 | sean-k-mooney | this basically recreates teh state to the vm before it started runing for the first tiem | |
| 13:11:21 | sean-k-mooney | we then do a blockdevie rebase on the delta disk to update it with the changes that have hppened since the vm booted. | |
| 13:12:04 | sean-k-mooney | we then freeze the guest filesystem and abort the rebase job | |
| 13:12:09 | sean-k-mooney | instead of commiting it | |
| 13:12:20 | sean-k-mooney | then unfreeze the filesystem | |
| 13:12:44 | sean-k-mooney | at this point the vm is back to runing form the orginal instance disk | |
| 13:13:09 | sean-k-mooney | and we have a copy of the filesytem changes in the delta disk | |
| 13:14:19 | sean-k-mooney | we then update the ownwershp of the delta disk so that its owned by nova and revert the guest xml back ot what it was before the snapshot | |
| 13:14:45 | sean-k-mooney | then we flaten the delta disk into the final format for uploading | |
| 13:14:53 | sean-k-mooney | and delete the delta disk | |
| 13:15:02 | sean-k-mooney | finally we upload the image to glance. | |
| 13:15:56 | sean-k-mooney | by creating the delta disk as an overlay of the backing file durint the snapshot we only need enough storage for the changes since the guest booted | |
| 13:16:22 | damnthem | sean-k-mooney: thank you. I think i understood where is difference in my case and why it's works irrational for me. | |
| 13:16:24 | sean-k-mooney | while we are doing the format convertion we need storage for the vm + deta disk + flattened image | |
| 13:17:03 | sean-k-mooney | and after the snap shot we are just back to the orginal disk | |
| 13:33:55 | damnthem | sean-k-mooney: to be short - we disabled used_cow_images, so backing stores disabled .So this whole image/snapshot juggling looks pointless from outside. Thank you for helping me figure this out | |
| 13:40:22 | sean-k-mooney | damnthem: so your using raw images | |
| 13:40:45 | sean-k-mooney | this is still useful in this case for older release as it reduces the time the mirror action takes | |
| 14:39:45 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Process unlimited exceptions raised by unplug_vifs https://review.opendev.org/c/openstack/nova/+/879350 | |
| 14:41:48 | auniyal | Hi dansmith | |
| 14:41:55 | dansmith | hi | |
| 14:41:56 | auniyal | regarding this: https://review.opendev.org/c/openstack/nova-specs/+/878757 | |
| 14:42:02 | auniyal | thanks for looking :) | |
| 14:42:11 | auniyal | as you said, now I too think the best way to update DB is with process of restarting instnace. | |
| 14:42:21 | auniyal | I was looking for the place where, we can updated DB, on instance shutoff. | |
| 14:42:33 | auniyal | I think it should be in compute/api as it has stop and start functionality. | |
| 14:42:50 | auniyal | I was also looking for list of operation nova might perform on instance shutdown or start, but could not find one. | |
| 14:43:02 | auniyal | Is this alright to add a decorator which perform all operations on instnace shutoff | |
| 14:43:15 | auniyal | something like | |
| 14:43:20 | auniyal | @post_shutoff_actions | |
| 14:43:20 | auniyal | def stop(instance): | |
| 14:43:20 | auniyal | pass | |
| 14:44:47 | dansmith | auniyal: compute/api is run on the caller, which means the api service would run that code for stop(), so no, I don't think that's best place | |
| 14:44:50 | dansmith | should be in manager | |
| 14:46:01 | auniyal | at this https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L3317 | |
| 15:02:58 | dansmith | auniyal: perhaps, but it probably would be better on start, but more like the lower level spawn so that it catches reboot, start, etc | |
| 15:05:05 | auniyal | dansmith, ack, | |
| 15:06:07 | auniyal | I didn't get, why we should not run DB update at caller (i.e controller as nova-api serice I believe !!) | |
| 15:06:25 | auniyal | is it because this action is DB related | |
| 15:08:15 | dansmith | no, for several reasons: | |
| 15:08:46 | dansmith | It needs to call to cinder and the database so it may be slow/blocking and holding the API caller while you do that is not good | |
| 15:09:09 | dansmith | it also means that it would only work for an actual api stop call and not for other things like reboot, or crash recovery, or guest-initiated reboot, etc | |
| 15:11:17 | auniyal | ack, got it, | |
| 15:37:30 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Process unlimited exceptions raised by unplug_vifs https://review.opendev.org/c/openstack/nova/+/879350 | |
| 15:46:58 | gibi | Uggla: I we we need https://review.opendev.org/c/openstack/nova/+/855664/ in train yes | |
| 15:47:18 | gibi | Uggla: is there a complication with the backport? | |
| 15:47:42 | Uggla | gibi, all ports should be upstream ? Any downstream to do ? | |
| 15:49:11 | Uggla | gibi, no I just try to check the "scope". | |
| 15:57:39 | gibi | Uggla: as we have train open upstream still we expected to land the fix upstream. | |
| 15:58:15 | gibi | if there is high pressure to get the fix earlier downstream then you can propose the downstream backport before the upstream backport lands, but we still need the upstream backport too | |
| 15:58:34 | gibi | (we probably discuss this downstream :D) | |
| 15:59:14 | Uggla | ok sounds good. | |
| 15:59:54 | gibi | Uggla: thanks for picking that fix up | |
| 16:00:32 | Uggla | gibi, you are welcome. | |
| 16:39:30 | gibi | :) | |
| #openstack-nova - 2023-04-04 | |||
| 02:09:15 | opendevreview | liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338 | |
| 02:15:20 | opendevreview | liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338 | |
| 04:45:37 | opendevreview | liang jiechao proposed openstack/nova-specs master: Generic vdpa spec https://review.opendev.org/c/openstack/nova-specs/+/879338 | |
| 08:38:32 | opendevreview | Konrad Gube proposed openstack/nova-specs master: Re-propose using extend volume completion action for 2023.2 https://review.opendev.org/c/openstack/nova-specs/+/877233 | |
| 08:50:40 | kgube | Hi bauzas, could you have a look at my spec? https://review.opendev.org/c/openstack/nova-specs/+/877233 | |
| 08:50:45 | kgube | You had some issues with the implemtentation for Antelope that might be better to discuss at the spec level: https://review.opendev.org/c/openstack/nova/+/873560 | |
| 08:50:58 | bauzas | kgube: ack, will try to do todaty | |
| 08:51:22 | kgube | thanks! | |
| 11:20:43 | sahid | lajoskatona: o/ if that sounds right for you I'm taking the lead to migrate nova on openstacksdk https://etherpad.opendev.org/p/python-neutronclient_deprecation | |
| 11:54:29 | lajoskatona | sahid: cool, thanks for checking it | |
| 12:23:30 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/xena: Add debug log for scheduler weight calculation https://review.opendev.org/c/openstack/nova/+/879404 | |
| 12:24:26 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/wallaby: Add debug log for scheduler weight calculation https://review.opendev.org/c/openstack/nova/+/879405 | |
| 13:33:08 | dansmith | bauzas: what regression do you think the long-wait change introduced? | |
| 13:33:38 | bauzas | dansmith: well, I don't really know but the test you modified was hit | |
| 13:34:03 | dansmith | uh, okay :) | |
| 13:36:34 | dansmith | bauzas: it's failing on create server, I only modified the attach volume line | |
| 13:36:41 | dansmith | so I think it's not likely related | |
| 13:36:44 | bauzas | ok | |
| 13:50:54 | wangrong | dansmith: hello Dan, based on our previous agreement, we have prepared the relevant spec and would like you to review them. If you have any questions, please contact with us anytime. Thank you! | |
| 13:50:54 | wangrong | https://review.opendev.org/c/openstack/nova-specs/+/877291 | |
| 13:51:41 | dansmith | wangrong: that's my spec that I showed you as an example.. did you paste the wrong link? | |
| 13:53:30 | wangrong | dansmith: oh, sorry, my bad... | |
| 13:53:47 | wangrong | dansmith: https://review.opendev.org/c/openstack/nova-specs/+/879338 | |
| 13:54:12 | wangrong | dansmith: I think this the one we posted | |
| 13:54:32 | dansmith | wangrong: cool, sean-k-mooney is the one that needs to do most of the review, but I'll try to take a first stab later today as well | |
| 13:54:49 | dansmith | bauzas: also cc ^ as discussed maybe you can provide some early feedback like we promised | |
| 13:54:58 | bauzas | cool | |
| 13:57:14 | wangrong | sure, thank you very much, so appreciate your help, dansmith, bauzas and sean-k-mooney . | |
| 14:02:45 | bauzas | as a reminder, due to European DST change one week before, the nova meeting will be in ~2 hours | |
| 14:03:13 | bauzas | (still the same UTC time, in case you're not in Europe) | |
| 15:11:19 | TheJulia | Are there any known issues on grenade from 2023.1 to master in regards to flavors? Specifically we're seeing a ~40% failure rate for grenade jobs on ironic, nova is specifically reporting the flavor baremetal is not found when it was added 5+ minutes earlier in the logs. Looks like post-create (200 response code), a lookup for it immediately fails with a 404. | |
| 15:11:19 | TheJulia | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b1d/879215/3/check/ironic-grenade/b1d6cf4/controller/logs/screen-n-api.txt | |
| 15:17:12 | TheJulia | and the returned flavor does indicate it is public, btw. | |
| 15:37:11 | sean-k-mooney | it might be related to srbac | |
| 15:37:39 | sean-k-mooney | tempest need to be using proejct scoped tokens | |