| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-01 | |||
| 12:29:43 | sean-k-mooney | once nova start the migration libvirt is basically in charge until its done | |
| 12:30:17 | sean-k-mooney | we can call libvirt with addtional commands like force complete or abort | |
| 12:31:11 | sean-k-mooney | but nova does not activly monitor the downtime and progress we passivly recive the events form libvirt and we can query it in resonce to an api request but we just get out of libvirts way and let it do its thing | |
| 12:34:15 | pmonteir | Oh... I see, so if the max downtime is hit, that's just something we let libvirt handle? | |
| 12:35:33 | sean-k-mooney | i can check, in the case of max im not sure but the normal pauses ectra are not done by nova | |
| 12:36:53 | sean-k-mooney | we configure the max down time on the guest https://github.com/openstack/nova/blob/master/nova/virt/libvirt/migration.py#L475-L527 | |
| 12:37:17 | sean-k-mooney | im just checkign to see if we handel when its exceeded in nova or preconfigure the action to take | |
| 12:38:25 | pmonteir | Yh, that's exactly the thing I'm having some trouble trying to find out. If something is actually done when its exceeded | |
| 12:39:23 | sean-k-mooney | i know we have action if the over all migration timeout is exceeded but not sure about max_downtime | |
| 12:39:32 | sean-k-mooney | kashyap: do you happen to know ^ | |
| 12:41:01 | kashyap | I don't remember off-hand, have to dig in too | |
| 12:42:00 | sean-k-mooney | pmonteir: what are you trying to achive by modifyign this by the way | |
| 12:42:10 | sean-k-mooney | pmonteir: i assume you have seen https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/doc/source/admin/configuring-migrations.rst#advanced-configuration-for-kvm-and-qemu | |
| 12:42:45 | kashyap | Yeah, let's step back to understand the bigger picture | |
| 12:43:15 | sean-k-mooney | the auto convergence and post copy options might be of interest to you if you are seeign excessive downtime | |
| 12:43:47 | sean-k-mooney | if you are seeing network downtime/ping loss that is likely unrealted to this | |
| 12:44:23 | pmonteir | Yh, I have. So basically I was trying to understand this parameter and test it out. But I wasn't able to see it in action (the max downtime value being exceeded and a timeout being triggered) | |
| 12:45:09 | sean-k-mooney | unless the vm is hevially loaded and dirtying memory you wont hit this in a normal migration | |
| 12:47:03 | pmonteir | I was trying to migrate whilst dirtying memory, the migration kept going for some time and (once) I used "virsh list" and the vm being migrated got paused and stayed like that for a while, which I found it weird | |
| 12:47:50 | sean-k-mooney | pmonteir: the migration timeout proably got exceeded and use force-compelte | |
| 12:48:14 | sean-k-mooney | https://github.com/openstack/nova/blob/0d84833e9688e0df97f3d24e06025e512bca3ce3/nova/conf/libvirt.py#L372-L388 | |
| 12:48:21 | sean-k-mooney | its our default | |
| 12:49:12 | sean-k-mooney | that is based on live_migration_completion_timeout not based on downtime and downtime has no effect on how long the vm will be paused | |
| 12:50:22 | sean-k-mooney | pmonteir: if that is what happened you should see https://github.com/openstack/nova/blob/2f644a82fec13bad8fcdfa195c9316a6f09ee15a/nova/virt/libvirt/migration.py#L468-L469 in the log | |
| 12:51:30 | kashyap | pmonteir: So, one more way to make sure libvirt is actually *setting* the thing by passing it to QEMU is to observe the libvirt/QEMU interaction logs | |
| 12:51:34 | kashyap | pmonteir: You can do it this way: | |
| 12:53:09 | kashyap | (1) Config the libvirt/QEMU log filters using the 'virt-admin' tool (it gets installed as part of "libvirt-daemon" package on Fedora; check for your distro): | |
| 12:53:14 | kashyap | $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 12:53:17 | kashyap | $> virt-admin daemon-log-filters "1:qemu_monitor" | |
| 12:53:59 | kashyap | (2) Migrate your guest from source to destination (and also make sure you've set the "_downtime" config parameter) | |
| 12:54:29 | kashyap | (3) That's it. Now you can `grep` for "downtime-limit" on source and destination libvirtd.log | |
| 12:54:45 | kashyap | Or ... you can post the output of this: grep -Ei '(Send Command|QEMU_MONITOR_RECV_)' /var/log/libvirt/libvirtd.log | |
| 12:55:04 | kashyap | In a paste-bin, I can analyze it. (Get the above output from both source and destination compute nodes) | |
| 12:55:22 | sean-k-mooney | gibi: im more or less +2+w on https://review.opendev.org/c/openstack/nova/+/792356/8//COMMIT_MSG but do you want me to hold off so that teh implemented ... can be added? | |
| 12:57:05 | gibi | sean-k-mooney: no, lets get it landed | |
| 12:57:49 | pmonteir | I will try config those logs and see what I can get. Thank you guys! | |
| 12:58:18 | sean-k-mooney | gibi: ok ill hit it when i get back just going to grab a drink | |
| 12:58:24 | gibi | ack | |
| 12:58:29 | dmitriis | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/826675/5/check/nova-next/3661dcb/testr_results.html | |
| 12:58:29 | dmitriis | sean-k-mooney: do you know if test_tagged_attachment is known to be flaky? Seem to be hitting it intermittently in both https://review.opendev.org/c/openstack/nova/+/819494 and https://review.opendev.org/c/openstack/nova/+/826675 | |
| 13:04:57 | kashyap | pmonteir: Also, a broader tip: whenever you're debugging something that involves libvirt and QEMU: the above logs with the said filters are useful to investigate. (Upstream DevStack actually logs these by default) | |
| 13:06:28 | sean-k-mooney | dmitriis: not that im aware of but volume attachment can be | |
| 13:06:42 | pmonteir | kashyap: Oh, thanks! I'm kinda noob to openstack/libvirt so this is really helpful! :D | |
| 13:07:20 | kashyap | pmonteir: No prob. The above logs are specifically good to debug all live migration issues | |
| 13:07:51 | pmonteir | got it! | |
| 13:08:56 | kashyap | pmonteir: Oh, wait. I had an error in the second step. It needs a bit more: | |
| 13:10:16 | kashyap | Not the second step, but the second command in the first step, for filters. The correct set of filters are: | |
| 13:10:16 | dmitriis | sean-k-mooney: ack, ty | |
| 13:10:20 | kashyap | $> virt-admin daemon-log-filters "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 13:10:37 | kashyap | pmonteir: --^ Make the above edit, if you took a note of this. | |
| 13:12:06 | pmonteir | oh, nice, thanks! | |
| 14:47:23 | gibi | bauzas: fyi I've filed a gate-failure bug https://bugs.launchpad.net/nova/+bug/1959677 it is happening daily | |
| 14:47:38 | bauzas | thanks, I was about to update the agenda | |
| 14:58:17 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833 | |
| 14:58:17 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | |
| 14:58:18 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Bump os-traits to 2.7.0 https://review.opendev.org/c/openstack/nova/+/826675 | |
| 14:58:18 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Introduce remote_managed tag for PCI devs https://review.opendev.org/c/openstack/nova/+/824834 | |
| 14:58:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | |
| 14:58:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | |
| 15:04:30 | dmitriis | sean-k-mooney, gibi: ta for the reviews. Fixed the first two, looking at gibi's comments to the third one. | |
| 15:05:24 | gibi | dmitriis: ack, I will check back at some point | |
| 15:15:48 | gibi | sean-k-mooney: when you read further into the smartnic series, I would like to see your opinion about the dynamicy capability handling at https://review.opendev.org/c/openstack/nova/+/812111/16/nova/compute/resource_tracker.py#1176 I have not problem with it but it is someting new | |
| 15:16:47 | sean-k-mooney | dynamicy capability handeling? | |
| 15:17:06 | opendevreview | Jonathan Race proposed openstack/nova master: Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 15:17:21 | artom | TypeError: can only concatenate str (not "int") to str" a known thing | |
| 15:17:21 | artom | Is "'name': old_flavor['name'] + 'extra-' + data_utils.rand_int_id(), | |
| 15:17:21 | sean-k-mooney | oh the traits reporting | |
| 15:17:27 | artom | Happening in nova-next | |
| 15:17:51 | sean-k-mooney | gibi: we have something simiarl i belive else where but slightly less clever in implemenation | |
| 15:17:56 | artom | Looks like a Tempest code error, perhaps data_utils.rand_int_id() changed to return an int or something, but I feel like if it was tempest it would be all over the place | |
| 15:18:24 | gibi | artom: that logic is a recent addition in tempest | |
| 15:18:52 | sean-k-mooney | artom: so ya that looks like it shoudl have a str() call | |
| 15:18:59 | artom | gibi, ah, right, cae966812a4a5070c3e7f82d16ebe697da57e5c5 | |
| 15:19:04 | artom | Well, it's breaking nova-next | |
| 15:19:04 | sean-k-mooney | althoguh personally i woudl be tempted to use an fstring | |
| 15:19:12 | gibi | yepp you were faster finding it | |
| 15:19:23 | gibi | artom: thanks | |
| 15:20:00 | sean-k-mooney | the fix is firly simple just use sting interpulation of fstring instead fo concationation or cast | |
| 15:20:01 | gibi | sean-k-mooney: cool | |
| 15:20:58 | sean-k-mooney | normally we do simple things like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L395-L420 | |
| 15:20:59 | gibi | artom: the orginal tempest fix did not trigger nova-next (or an ovs full tempest from neutron) and therefore the test case that is changed was not run for the tempest patch | |
| 15:21:19 | artom | Yeah, all good. Quick enough fix | |
| 15:21:30 | artom | It'll take me longer to file the LP bug than changing the code :P | |
| 15:21:41 | sean-k-mooney | but we also build trait dymically like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8412-L8434 | |
| 15:21:52 | gibi | artom: please add a DNM patch in nova that depends on the tempest fix to see nova-next is green after it :) | |
| 15:21:58 | sean-k-mooney | gibi: is your concern that htis is a mor dynimic capblity | |
| 15:22:03 | sean-k-mooney | not just a dynmic trait | |
| 15:22:04 | artom | gibi, good idea | |
| 15:22:15 | gibi | sean-k-mooney: this is a new thing and it will create precedent | |
| 15:22:32 | gibi | sean-k-mooney: I mean that we dinamically apply capability based on resource inventory | |
| 15:23:19 | gibi | sean-k-mooney: so I'd like to make a conscious introducing it | |
| 15:23:21 | sean-k-mooney | oh as in it only reports the trait when we have a pool that support it | |
| 15:23:26 | gibi | yepp | |
| 15:23:41 | sean-k-mooney | right | |
| 15:23:50 | gibi | but personally I have no problem with it | |
| 15:24:01 | sean-k-mooney | well we technially dont need to do that actully | |
| 15:24:05 | sean-k-mooney | i dont really either | |
| 15:24:16 | sean-k-mooney | but we coudl jsut always report it | |
| 15:24:20 | sean-k-mooney | if the host support it | |
| 15:24:34 | gibi | it is basically indirectly config driver (by the whitelist config) and we already have config driven capabilities | |