| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-01 | |||
| 12:41:01 | kashyap | I don't remember off-hand, have to dig in too | |
| 12:42:00 | sean-k-mooney | pmonteir: what are you trying to achive by modifyign this by the way | |
| 12:42:10 | sean-k-mooney | pmonteir: i assume you have seen https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/doc/source/admin/configuring-migrations.rst#advanced-configuration-for-kvm-and-qemu | |
| 12:42:45 | kashyap | Yeah, let's step back to understand the bigger picture | |
| 12:43:15 | sean-k-mooney | the auto convergence and post copy options might be of interest to you if you are seeign excessive downtime | |
| 12:43:47 | sean-k-mooney | if you are seeing network downtime/ping loss that is likely unrealted to this | |
| 12:44:23 | pmonteir | Yh, I have. So basically I was trying to understand this parameter and test it out. But I wasn't able to see it in action (the max downtime value being exceeded and a timeout being triggered) | |
| 12:45:09 | sean-k-mooney | unless the vm is hevially loaded and dirtying memory you wont hit this in a normal migration | |
| 12:47:03 | pmonteir | I was trying to migrate whilst dirtying memory, the migration kept going for some time and (once) I used "virsh list" and the vm being migrated got paused and stayed like that for a while, which I found it weird | |
| 12:47:50 | sean-k-mooney | pmonteir: the migration timeout proably got exceeded and use force-compelte | |
| 12:48:14 | sean-k-mooney | https://github.com/openstack/nova/blob/0d84833e9688e0df97f3d24e06025e512bca3ce3/nova/conf/libvirt.py#L372-L388 | |
| 12:48:21 | sean-k-mooney | its our default | |
| 12:49:12 | sean-k-mooney | that is based on live_migration_completion_timeout not based on downtime and downtime has no effect on how long the vm will be paused | |
| 12:50:22 | sean-k-mooney | pmonteir: if that is what happened you should see https://github.com/openstack/nova/blob/2f644a82fec13bad8fcdfa195c9316a6f09ee15a/nova/virt/libvirt/migration.py#L468-L469 in the log | |
| 12:51:30 | kashyap | pmonteir: So, one more way to make sure libvirt is actually *setting* the thing by passing it to QEMU is to observe the libvirt/QEMU interaction logs | |
| 12:51:34 | kashyap | pmonteir: You can do it this way: | |
| 12:53:09 | kashyap | (1) Config the libvirt/QEMU log filters using the 'virt-admin' tool (it gets installed as part of "libvirt-daemon" package on Fedora; check for your distro): | |
| 12:53:14 | kashyap | $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 12:53:17 | kashyap | $> virt-admin daemon-log-filters "1:qemu_monitor" | |
| 12:53:59 | kashyap | (2) Migrate your guest from source to destination (and also make sure you've set the "_downtime" config parameter) | |
| 12:54:29 | kashyap | (3) That's it. Now you can `grep` for "downtime-limit" on source and destination libvirtd.log | |
| 12:54:45 | kashyap | Or ... you can post the output of this: grep -Ei '(Send Command|QEMU_MONITOR_RECV_)' /var/log/libvirt/libvirtd.log | |
| 12:55:04 | kashyap | In a paste-bin, I can analyze it. (Get the above output from both source and destination compute nodes) | |
| 12:55:22 | sean-k-mooney | gibi: im more or less +2+w on https://review.opendev.org/c/openstack/nova/+/792356/8//COMMIT_MSG but do you want me to hold off so that teh implemented ... can be added? | |
| 12:57:05 | gibi | sean-k-mooney: no, lets get it landed | |
| 12:57:49 | pmonteir | I will try config those logs and see what I can get. Thank you guys! | |
| 12:58:18 | sean-k-mooney | gibi: ok ill hit it when i get back just going to grab a drink | |
| 12:58:24 | gibi | ack | |
| 12:58:29 | dmitriis | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/826675/5/check/nova-next/3661dcb/testr_results.html | |
| 12:58:29 | dmitriis | sean-k-mooney: do you know if test_tagged_attachment is known to be flaky? Seem to be hitting it intermittently in both https://review.opendev.org/c/openstack/nova/+/819494 and https://review.opendev.org/c/openstack/nova/+/826675 | |
| 13:04:57 | kashyap | pmonteir: Also, a broader tip: whenever you're debugging something that involves libvirt and QEMU: the above logs with the said filters are useful to investigate. (Upstream DevStack actually logs these by default) | |
| 13:06:28 | sean-k-mooney | dmitriis: not that im aware of but volume attachment can be | |
| 13:06:42 | pmonteir | kashyap: Oh, thanks! I'm kinda noob to openstack/libvirt so this is really helpful! :D | |
| 13:07:20 | kashyap | pmonteir: No prob. The above logs are specifically good to debug all live migration issues | |
| 13:07:51 | pmonteir | got it! | |
| 13:08:56 | kashyap | pmonteir: Oh, wait. I had an error in the second step. It needs a bit more: | |
| 13:10:16 | kashyap | Not the second step, but the second command in the first step, for filters. The correct set of filters are: | |
| 13:10:16 | dmitriis | sean-k-mooney: ack, ty | |
| 13:10:20 | kashyap | $> virt-admin daemon-log-filters "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 13:10:37 | kashyap | pmonteir: --^ Make the above edit, if you took a note of this. | |
| 13:12:06 | pmonteir | oh, nice, thanks! | |
| 14:47:23 | gibi | bauzas: fyi I've filed a gate-failure bug https://bugs.launchpad.net/nova/+bug/1959677 it is happening daily | |
| 14:47:38 | bauzas | thanks, I was about to update the agenda | |
| 14:58:17 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833 | |
| 14:58:17 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | |
| 14:58:18 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Bump os-traits to 2.7.0 https://review.opendev.org/c/openstack/nova/+/826675 | |
| 14:58:18 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Introduce remote_managed tag for PCI devs https://review.opendev.org/c/openstack/nova/+/824834 | |
| 14:58:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | |
| 14:58:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | |
| 15:04:30 | dmitriis | sean-k-mooney, gibi: ta for the reviews. Fixed the first two, looking at gibi's comments to the third one. | |
| 15:05:24 | gibi | dmitriis: ack, I will check back at some point | |
| 15:15:48 | gibi | sean-k-mooney: when you read further into the smartnic series, I would like to see your opinion about the dynamicy capability handling at https://review.opendev.org/c/openstack/nova/+/812111/16/nova/compute/resource_tracker.py#1176 I have not problem with it but it is someting new | |
| 15:16:47 | sean-k-mooney | dynamicy capability handeling? | |
| 15:17:06 | opendevreview | Jonathan Race proposed openstack/nova master: Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 15:17:21 | artom | TypeError: can only concatenate str (not "int") to str" a known thing | |
| 15:17:21 | artom | Is "'name': old_flavor['name'] + 'extra-' + data_utils.rand_int_id(), | |
| 15:17:21 | sean-k-mooney | oh the traits reporting | |
| 15:17:27 | artom | Happening in nova-next | |
| 15:17:51 | sean-k-mooney | gibi: we have something simiarl i belive else where but slightly less clever in implemenation | |
| 15:17:56 | artom | Looks like a Tempest code error, perhaps data_utils.rand_int_id() changed to return an int or something, but I feel like if it was tempest it would be all over the place | |
| 15:18:24 | gibi | artom: that logic is a recent addition in tempest | |
| 15:18:52 | sean-k-mooney | artom: so ya that looks like it shoudl have a str() call | |
| 15:18:59 | artom | gibi, ah, right, cae966812a4a5070c3e7f82d16ebe697da57e5c5 | |
| 15:19:04 | artom | Well, it's breaking nova-next | |
| 15:19:04 | sean-k-mooney | althoguh personally i woudl be tempted to use an fstring | |
| 15:19:12 | gibi | yepp you were faster finding it | |
| 15:19:23 | gibi | artom: thanks | |
| 15:20:00 | sean-k-mooney | the fix is firly simple just use sting interpulation of fstring instead fo concationation or cast | |
| 15:20:01 | gibi | sean-k-mooney: cool | |
| 15:20:58 | sean-k-mooney | normally we do simple things like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L395-L420 | |
| 15:20:59 | gibi | artom: the orginal tempest fix did not trigger nova-next (or an ovs full tempest from neutron) and therefore the test case that is changed was not run for the tempest patch | |
| 15:21:19 | artom | Yeah, all good. Quick enough fix | |
| 15:21:30 | artom | It'll take me longer to file the LP bug than changing the code :P | |
| 15:21:41 | sean-k-mooney | but we also build trait dymically like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8412-L8434 | |
| 15:21:52 | gibi | artom: please add a DNM patch in nova that depends on the tempest fix to see nova-next is green after it :) | |
| 15:21:58 | sean-k-mooney | gibi: is your concern that htis is a mor dynimic capblity | |
| 15:22:03 | sean-k-mooney | not just a dynmic trait | |
| 15:22:04 | artom | gibi, good idea | |
| 15:22:15 | gibi | sean-k-mooney: this is a new thing and it will create precedent | |
| 15:22:32 | gibi | sean-k-mooney: I mean that we dinamically apply capability based on resource inventory | |
| 15:23:19 | gibi | sean-k-mooney: so I'd like to make a conscious introducing it | |
| 15:23:21 | sean-k-mooney | oh as in it only reports the trait when we have a pool that support it | |
| 15:23:26 | gibi | yepp | |
| 15:23:41 | sean-k-mooney | right | |
| 15:23:50 | gibi | but personally I have no problem with it | |
| 15:24:01 | sean-k-mooney | well we technially dont need to do that actully | |
| 15:24:05 | sean-k-mooney | i dont really either | |
| 15:24:16 | sean-k-mooney | but we coudl jsut always report it | |
| 15:24:20 | sean-k-mooney | if the host support it | |
| 15:24:34 | gibi | it is basically indirectly config driver (by the whitelist config) and we already have config driven capabilities | |
| 15:24:37 | sean-k-mooney | the inventory check is going to be enforce by the pci filter after the fact | |
| 15:24:48 | gibi | s/driver/driven/ | |
| 15:24:50 | sean-k-mooney | yes we do | |
| 15:25:07 | sean-k-mooney | the main advantage to having it is the filtering will partly be done by placment | |
| 15:25:44 | gibi | yepp | |
| 15:25:47 | gibi | that is a good thing | |
| 15:25:59 | sean-k-mooney | the smart nics port dont have inventories in placement currntly but this will at least find host with them in the resouce tracker | |
| 15:26:25 | sean-k-mooney | so ya in the long run i agree i think this better and also that yes this is a new thing and setting precident | |
| 15:26:46 | gibi | awesome, then we are on the same page | |
| 15:27:24 | sean-k-mooney | am since we are talking about pci i agree with you regfarding the pack/spread change. i raised the behaivor delta in a prevous version | |