| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-05-22 | |||
| 05:29:47 | sean-k-mooney | yep | |
| 05:30:16 | sean-k-mooney | its proably a good weekend hackign project which is also where we shoudl put it | |
| 05:30:30 | sean-k-mooney | in the hacking repo so we can share it acrross other repos | |
| 05:30:33 | mikal | It would also make the transition largely transparent... People do a git review, land a new version of the hook automatically, and blam they're compliant. Much more fun than herding cats. | |
| 05:31:09 | sean-k-mooney | we can add a new hook there and then enable it in the relevent repos https://github.com/openstack/hacking/blob/master/.pre-commit-hooks.yaml | |
| 05:31:42 | mikal | Although... Other projects also use signed-off-by to track who reviewed and approved a change. Is there any intent to do that as well? | |
| 05:34:25 | sean-k-mooney | mikal: that actully a missues of it | |
| 05:34:36 | sean-k-mooney | mikal: there are other standared lables for that | |
| 05:35:04 | sean-k-mooney | mikal: so no not currently however you can actully config gerrit to auto ammend the commit with that info to some degree | |
| 05:35:24 | sean-k-mooney | we had that enabeld downstream untile they moved use to gitlab..... | |
| 05:35:41 | mikal | I kind of like the history being all in one place to be honest. | |
| 05:36:30 | mikal | So yeah, review history in the commit would be cool. I don't really care what labels are used to be honest. I do agree the kernel seems to use "Signed-off-by", "Tested-by", "Reviewed-by", etc. | |
| 05:37:36 | sean-k-mooney | your ment to use Reviewed-by, Acked-by and Tested-by instead of sgined-off by to track that | |
| 05:38:33 | sean-k-mooney | so you coudl update the comit with the review vots and add acted by for the person that +w'd and tested-by by zuul (the verifed label votes) | |
| 05:39:17 | sean-k-mooney | however to do that gerrit would have to be sett to cerry pick mode which will alwasy cheryy pick the commit on to the target branch changign it git sha | |
| 05:39:38 | sean-k-mooney | which will mean you will not be able to knwo the sha to use for backportign until after it has merged because it will alwasy change | |
| 05:40:10 | sean-k-mooney | so to do that we loose our current cherry-picked form workflow/lines when backporting | |
| 05:42:49 | sean-k-mooney | mikal: the convetions for the by lines alll come form teh linke kernel and are part of there patch submission process https://github.com/torvalds/linux/blob/master/Documentation/process/submitting-patches.rst#using-reported-by-tested-by-reviewed-by-suggested-by-and-fixes | |
| 07:07:37 | sean-k-mooney | bauzas: just so your aware i rebased your mtty_suppot sersie and moved my patch to the bottom. | |
| 07:08:06 | sean-k-mooney | bauzas: with it i was able to fix the error you were hitting but there is a bug in your first patch https://review.opendev.org/c/openstack/nova/+/898100/11 | |
| 07:08:29 | sean-k-mooney | the error i got with my fix was libvirt.libvirtError: Requested operation is not valid: mediated device /sys/bus/mdev/devices/6dc8d190-2bc9-45ed-a4b8-d4f0708f92bb is in use by driver QEMU, domain instance-00000001 | |
| 07:09:56 | sean-k-mooney | so somehow nova tired to reuse the same mdev between to vms. i assuem that is a bug in https://review.opendev.org/c/openstack/nova/+/898100/11 that only affect virutal mdev device if not that woudl be a real bug in our current mdev supprot | |
| 07:10:26 | sean-k-mooney | i suspect it has somthign to do wtth how your trackign the virutal ones given they do not have a pci adress but im not sure | |
| 07:27:32 | mikal | sean-k-mooney: when you have a moment, I feel like the state of the VDI patches has fallen out of my brain. Are you now happy enough with the first two patches in the series that I could squash them into one and then do the compute protection change to the other set of extra specs? | |
| 07:29:44 | sean-k-mooney | i share your oxidation. ill need to quickly load context | |
| 07:30:11 | sean-k-mooney | you asking about squashing https://review.opendev.org/c/openstack/nova/+/940770/9 and the patch before it | |
| 07:30:14 | sean-k-mooney | or all 3 | |
| 07:31:19 | sean-k-mooney | mikal: i know dan prefers if we sepreate the object change form the driver change | |
| 07:32:12 | sean-k-mooney | so i think there overall prefence woudl be have the first patch containe the object and noticicaton object change. then move the rest of the changes in that to the "Protect older compute managers from sound model requests." change | |
| 07:32:18 | mikal | So at the moment the chain is: sound device extra spec -> compute manager protection for sound -> usb controller extra spec -> compute manager protection for USB (this last one not yet done) | |
| 07:32:43 | mikal | I had thought you'd want the first two squashed so you get the extra spec and the compute protection at the same time. | |
| 07:33:11 | sean-k-mooney | yes i want that btu we geneally prefer to have the object chagns in there won patch | |
| 07:33:24 | sean-k-mooney | so it would still be 4 patchees | |
| 07:33:27 | mikal | I think you're saying for those first two you'd instead prefer to see the first to squashed into one, which becomes the second patch, with the object and notification changes sitting before it? | |
| 07:33:53 | mikal | So: sound objs and notifications -> rest of sound -> usb objs and notifications -> rest of usb? | |
| 07:34:03 | sean-k-mooney | "add object changes for sound" -> "support soudn device" -> "add object changes for usb" "supprot usb" | |
| 07:34:09 | sean-k-mooney | yes | |
| 07:34:39 | mikal | Cool. I can do that. Are you happy with the content of the first two patches, just not the way they're presented? Or are there other changes that need to be made too? | |
| 07:35:02 | sean-k-mooney | mikal: in the unlikely case we need to revert this its easy to rever the driver change but we can really revert the object change | |
| 07:35:17 | sean-k-mooney | so having it be seperate just makes taht edge case or backport a littel simpler | |
| 07:35:47 | mikal | sean-k-mooney: that's fair. I'll do that refactor ASAPish if you're otherwise happy with the code. | |
| 07:36:39 | sean-k-mooney | ill need to look at the detail but i think i can review this series today | |
| 07:36:50 | sean-k-mooney | i dont think there was anything major last time i looked | |
| 07:37:05 | sean-k-mooney | so i dont really expect there to be any thing that need to be changed | |
| 07:37:09 | mikal | Yeah, certainly I can't see anything major in the comments unless I've missed something. | |
| 07:37:29 | mikal | Ok cool. I will do the refactor then and implement the compute manager protection for USB as well. | |
| 09:23:25 | noonedeadpunk | Hey folks! Got a question. Today realized that quite an unexpected behaviour is happening with live migrations. So on memory-intense VMs migration seems to fail on post-migration step. | |
| 09:23:57 | noonedeadpunk | The weird/enexpected thing, that as a result VM keeps shut down on both hosts and it moves to ERROR state | |
| 09:24:22 | noonedeadpunk | so decided to ask if this is smth quite well known? | |
| 09:24:44 | noonedeadpunk | As right now I think it's related to `live_migration_permit_auto_converge = true` | |
| 09:25:32 | noonedeadpunk | as issue happens only on memory intense VMs. So if you run stress inside VM - it will 99% end up in ERROR and shut down. If there's some reasonable applucation running - it's fine | |
| 09:26:17 | noonedeadpunk | So my guess is that if live_migration_permit_auto_converge kicks in, libvirt simply incapable to resume the VM on thye destination, but I'm not really sure if that is true | |
| 09:26:57 | noonedeadpunk | Thus before going to this rabbit hole wanted to ask if you might now anything about that :) | |
| 09:30:15 | frickler | noonedeadpunk: I assume you checked the nova-compute logs without success? how about the libvirt instance logs? but maybe wait for sean-k-mooney to chime in first | |
| 09:31:00 | sean-k-mooney | sorry was looking at something else reading back | |
| 09:32:09 | sean-k-mooney | noonedeadpunk: do you have post_copy migration enabeld? | |
| 09:32:44 | sean-k-mooney | auto converge just add cpu puases so that is very unlikely to be a factor | |
| 09:34:37 | sean-k-mooney | noonedeadpunk: for what its worth if there is high ram load it basically imposibel to migrate without post-copy migration. auto converge si not entough to supprot that usecase | |
| 09:34:52 | sean-k-mooney | it just slightly chagne wehre the live mgiration will fall over | |
| 09:36:00 | sean-k-mooney | auto converge is still a type of "pre-copy" migration when qemu copes all the memroy first , pauses the vm on the dest copyies the detal and resuses it on the souce | |
| 09:36:12 | noonedeadpunk | no, post_copy is disabled | |
| 09:36:16 | noonedeadpunk | only converge is | |
| 09:36:47 | sean-k-mooney | the difference betwen live_migration_permit_auto_converge = true and false is if its set to true. the qemu on the souce is allwo to micro pause the souce vm cpus before switching to the dest | |
| 09:37:10 | noonedeadpunk | yes, right, and that used to work nicely | |
| 09:37:27 | sean-k-mooney | in your case if your getting to post_live_migration that mean libvirt told nova the migration was complete | |
| 09:37:29 | noonedeadpunk | and I think regardless that should not end up with VM jsut being down on both sides? | |
| 09:37:45 | noonedeadpunk | jsut a sec, I'll re-paste logs | |
| 09:38:03 | sean-k-mooney | at which point the vm on the souce is not need any mroe and shoudl be terminate by libvirt | |
| 09:38:10 | sean-k-mooney | as you cant rool back at that point | |
| 09:38:18 | noonedeadpunk | it;s from source https://paste.openstack.org/show/bSc55xr9LSgCXiaj1uo1/ | |
| 09:38:33 | noonedeadpunk | and that's destination: https://paste.openstack.org/show/boBhuqrR8mYDoqCjYxex/ | |
| 09:39:01 | noonedeadpunk | the only very suspicious thing I've spotted, is that I also do see a libvirt reseted communication to qemu | |
| 09:40:25 | sean-k-mooney | i knwo you said this but the vm is activly runing right it not paused when you live migrated it right | |
| 09:40:30 | noonedeadpunk | so it feels that during unpause things crash unobviously | |
| 09:40:41 | noonedeadpunk | it's running, yes | |
| 09:40:46 | noonedeadpunk | I mean paused on destination | |
| 09:41:22 | sean-k-mooney | the reaons im asking is this is sort of similar to https://issues.redhat.com/browse/RHEL-48801 but that about live migrating a puased instance twice | |
| 09:41:59 | sean-k-mooney | qemu was not probly handling something in the block storage layer in that case | |
| 09:42:00 | noonedeadpunk | there's also no block migration as storage is ceph | |
| 09:42:10 | sean-k-mooney | ack that also good to know | |
| 09:43:03 | noonedeadpunk | but yeah, trying out post_copy was my next step to try | |
| 09:43:15 | noonedeadpunk | Was jsut a bit afraid of it tbh | |
| 09:43:20 | sean-k-mooney | noonedeadpunk: can you check the qemu instance logs and see if there is an error on the source or dest | |
| 09:43:36 | sean-k-mooney | for the block device one we saw a message like this | |
| 09:43:39 | sean-k-mooney | qemu-kvm: ../block.c:6979: int bdrv_inactivate_recurse(BlockDriverState *): Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed. | |
| 09:43:41 | sean-k-mooney | 2024-07-16 13:29:55.795+0000: shutting down, reason=crashed | |
| 09:44:30 | sean-k-mooney | noonedeadpunk: post-copy is more robust to activly loaded vms and is effectivly requried if your using 1G hugepages | |
| 09:45:04 | sean-k-mooney | you can kind of get vms with 2M hugepages to migate without it but 1G hugepages guests are very hard to make function | |
| 09:45:32 | noonedeadpunk | I did not configure hugepages explicitly tbh | |
| 09:45:32 | sean-k-mooney | it woudl be woth trying but i suspect your hiting a qemu but and post copy might help you work around that but im not sure | |
| 09:45:49 | sean-k-mooney | noonedeadpunk: if you didnt then its not a factor | |
| 09:45:59 | sean-k-mooney | the kernel might be using transparent hugepages | |
| 09:46:06 | sean-k-mooney | but if that the case it wont impact the migration | |
| 09:46:19 | noonedeadpunk | yeah, let me try to reproduce one more time as original VM was dropped apparently | |
| 09:46:23 | sean-k-mooney | the issue with explcit hugepages is if you dirty the page qemu has to copy the entire page again | |
| 09:46:42 | sean-k-mooney | mean a 1 byte write requrie 1G of data to be copied again | |
| 09:47:42 | noonedeadpunk | so what would be suggested value? 1G? | |
| 09:48:03 | sean-k-mooney | sorry can you say that again | |
| 09:48:27 | sean-k-mooney | are you askign what i recomemnd if you choosoe to use hugepages? | |
| 09:48:48 | noonedeadpunk | Yeah :) I';d need to read about them more, but maybe you have a suggestion from your experience | |