| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-01 | |||
| 12:42:10 | sean-k-mooney | pmonteir: i assume you have seen https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/doc/source/admin/configuring-migrations.rst#advanced-configuration-for-kvm-and-qemu | |
| 12:42:45 | kashyap | Yeah, let's step back to understand the bigger picture | |
| 12:43:15 | sean-k-mooney | the auto convergence and post copy options might be of interest to you if you are seeign excessive downtime | |
| 12:43:47 | sean-k-mooney | if you are seeing network downtime/ping loss that is likely unrealted to this | |
| 12:44:23 | pmonteir | Yh, I have. So basically I was trying to understand this parameter and test it out. But I wasn't able to see it in action (the max downtime value being exceeded and a timeout being triggered) | |
| 12:45:09 | sean-k-mooney | unless the vm is hevially loaded and dirtying memory you wont hit this in a normal migration | |
| 12:47:03 | pmonteir | I was trying to migrate whilst dirtying memory, the migration kept going for some time and (once) I used "virsh list" and the vm being migrated got paused and stayed like that for a while, which I found it weird | |
| 12:47:50 | sean-k-mooney | pmonteir: the migration timeout proably got exceeded and use force-compelte | |
| 12:48:14 | sean-k-mooney | https://github.com/openstack/nova/blob/0d84833e9688e0df97f3d24e06025e512bca3ce3/nova/conf/libvirt.py#L372-L388 | |
| 12:48:21 | sean-k-mooney | its our default | |
| 12:49:12 | sean-k-mooney | that is based on live_migration_completion_timeout not based on downtime and downtime has no effect on how long the vm will be paused | |
| 12:50:22 | sean-k-mooney | pmonteir: if that is what happened you should see https://github.com/openstack/nova/blob/2f644a82fec13bad8fcdfa195c9316a6f09ee15a/nova/virt/libvirt/migration.py#L468-L469 in the log | |
| 12:51:30 | kashyap | pmonteir: So, one more way to make sure libvirt is actually *setting* the thing by passing it to QEMU is to observe the libvirt/QEMU interaction logs | |
| 12:51:34 | kashyap | pmonteir: You can do it this way: | |
| 12:53:09 | kashyap | (1) Config the libvirt/QEMU log filters using the 'virt-admin' tool (it gets installed as part of "libvirt-daemon" package on Fedora; check for your distro): | |
| 12:53:14 | kashyap | $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 12:53:17 | kashyap | $> virt-admin daemon-log-filters "1:qemu_monitor" | |
| 12:53:59 | kashyap | (2) Migrate your guest from source to destination (and also make sure you've set the "_downtime" config parameter) | |
| 12:54:29 | kashyap | (3) That's it. Now you can `grep` for "downtime-limit" on source and destination libvirtd.log | |
| 12:54:45 | kashyap | Or ... you can post the output of this: grep -Ei '(Send Command|QEMU_MONITOR_RECV_)' /var/log/libvirt/libvirtd.log | |
| 12:55:04 | kashyap | In a paste-bin, I can analyze it. (Get the above output from both source and destination compute nodes) | |
| 12:55:22 | sean-k-mooney | gibi: im more or less +2+w on https://review.opendev.org/c/openstack/nova/+/792356/8//COMMIT_MSG but do you want me to hold off so that teh implemented ... can be added? | |
| 12:57:05 | gibi | sean-k-mooney: no, lets get it landed | |
| 12:57:49 | pmonteir | I will try config those logs and see what I can get. Thank you guys! | |
| 12:58:18 | sean-k-mooney | gibi: ok ill hit it when i get back just going to grab a drink | |
| 12:58:24 | gibi | ack | |
| 12:58:29 | dmitriis | sean-k-mooney: do you know if test_tagged_attachment is known to be flaky? Seem to be hitting it intermittently in both https://review.opendev.org/c/openstack/nova/+/819494 and https://review.opendev.org/c/openstack/nova/+/826675 | |
| 12:58:29 | dmitriis | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/826675/5/check/nova-next/3661dcb/testr_results.html | |
| 13:04:57 | kashyap | pmonteir: Also, a broader tip: whenever you're debugging something that involves libvirt and QEMU: the above logs with the said filters are useful to investigate. (Upstream DevStack actually logs these by default) | |
| 13:06:28 | sean-k-mooney | dmitriis: not that im aware of but volume attachment can be | |
| 13:06:42 | pmonteir | kashyap: Oh, thanks! I'm kinda noob to openstack/libvirt so this is really helpful! :D | |
| 13:07:20 | kashyap | pmonteir: No prob. The above logs are specifically good to debug all live migration issues | |
| 13:07:51 | pmonteir | got it! | |
| 13:08:56 | kashyap | pmonteir: Oh, wait. I had an error in the second step. It needs a bit more: | |
| 13:10:16 | dmitriis | sean-k-mooney: ack, ty | |
| 13:10:16 | kashyap | Not the second step, but the second command in the first step, for filters. The correct set of filters are: | |
| 13:10:20 | kashyap | $> virt-admin daemon-log-filters "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 13:10:37 | kashyap | pmonteir: --^ Make the above edit, if you took a note of this. | |
| 13:12:06 | pmonteir | oh, nice, thanks! | |
| 14:47:23 | gibi | bauzas: fyi I've filed a gate-failure bug https://bugs.launchpad.net/nova/+bug/1959677 it is happening daily | |
| 14:47:38 | bauzas | thanks, I was about to update the agenda | |
| 14:58:17 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | |
| 14:58:17 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833 | |
| 14:58:18 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Introduce remote_managed tag for PCI devs https://review.opendev.org/c/openstack/nova/+/824834 | |
| 14:58:18 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Bump os-traits to 2.7.0 https://review.opendev.org/c/openstack/nova/+/826675 | |
| 14:58:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | |
| 14:58:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | |
| 15:04:30 | dmitriis | sean-k-mooney, gibi: ta for the reviews. Fixed the first two, looking at gibi's comments to the third one. | |
| 15:05:24 | gibi | dmitriis: ack, I will check back at some point | |
| 15:15:48 | gibi | sean-k-mooney: when you read further into the smartnic series, I would like to see your opinion about the dynamicy capability handling at https://review.opendev.org/c/openstack/nova/+/812111/16/nova/compute/resource_tracker.py#1176 I have not problem with it but it is someting new | |
| 15:16:47 | sean-k-mooney | dynamicy capability handeling? | |
| 15:17:06 | opendevreview | Jonathan Race proposed openstack/nova master: Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 15:17:21 | sean-k-mooney | oh the traits reporting | |
| 15:17:21 | artom | Is "'name': old_flavor['name'] + 'extra-' + data_utils.rand_int_id(), | |
| 15:17:21 | artom | TypeError: can only concatenate str (not "int") to str" a known thing | |
| 15:17:27 | artom | Happening in nova-next | |
| 15:17:51 | sean-k-mooney | gibi: we have something simiarl i belive else where but slightly less clever in implemenation | |
| 15:17:56 | artom | Looks like a Tempest code error, perhaps data_utils.rand_int_id() changed to return an int or something, but I feel like if it was tempest it would be all over the place | |
| 15:18:24 | gibi | artom: that logic is a recent addition in tempest | |
| 15:18:52 | sean-k-mooney | artom: so ya that looks like it shoudl have a str() call | |
| 15:18:59 | artom | gibi, ah, right, cae966812a4a5070c3e7f82d16ebe697da57e5c5 | |
| 15:19:04 | sean-k-mooney | althoguh personally i woudl be tempted to use an fstring | |
| 15:19:04 | artom | Well, it's breaking nova-next | |
| 15:19:12 | gibi | yepp you were faster finding it | |
| 15:19:23 | gibi | artom: thanks | |
| 15:20:00 | sean-k-mooney | the fix is firly simple just use sting interpulation of fstring instead fo concationation or cast | |
| 15:20:01 | gibi | sean-k-mooney: cool | |
| 15:20:58 | sean-k-mooney | normally we do simple things like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L395-L420 | |
| 15:20:59 | gibi | artom: the orginal tempest fix did not trigger nova-next (or an ovs full tempest from neutron) and therefore the test case that is changed was not run for the tempest patch | |
| 15:21:19 | artom | Yeah, all good. Quick enough fix | |
| 15:21:30 | artom | It'll take me longer to file the LP bug than changing the code :P | |
| 15:21:41 | sean-k-mooney | but we also build trait dymically like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8412-L8434 | |
| 15:21:52 | gibi | artom: please add a DNM patch in nova that depends on the tempest fix to see nova-next is green after it :) | |
| 15:21:58 | sean-k-mooney | gibi: is your concern that htis is a mor dynimic capblity | |
| 15:22:03 | sean-k-mooney | not just a dynmic trait | |
| 15:22:04 | artom | gibi, good idea | |
| 15:22:15 | gibi | sean-k-mooney: this is a new thing and it will create precedent | |
| 15:22:32 | gibi | sean-k-mooney: I mean that we dinamically apply capability based on resource inventory | |
| 15:23:19 | gibi | sean-k-mooney: so I'd like to make a conscious introducing it | |
| 15:23:21 | sean-k-mooney | oh as in it only reports the trait when we have a pool that support it | |
| 15:23:26 | gibi | yepp | |
| 15:23:41 | sean-k-mooney | right | |
| 15:23:50 | gibi | but personally I have no problem with it | |
| 15:24:01 | sean-k-mooney | well we technially dont need to do that actully | |
| 15:24:05 | sean-k-mooney | i dont really either | |
| 15:24:16 | sean-k-mooney | but we coudl jsut always report it | |
| 15:24:20 | sean-k-mooney | if the host support it | |
| 15:24:34 | gibi | it is basically indirectly config driver (by the whitelist config) and we already have config driven capabilities | |
| 15:24:37 | sean-k-mooney | the inventory check is going to be enforce by the pci filter after the fact | |
| 15:24:48 | gibi | s/driver/driven/ | |
| 15:24:50 | sean-k-mooney | yes we do | |
| 15:25:07 | sean-k-mooney | the main advantage to having it is the filtering will partly be done by placment | |
| 15:25:44 | gibi | yepp | |
| 15:25:47 | gibi | that is a good thing | |
| 15:25:59 | sean-k-mooney | the smart nics port dont have inventories in placement currntly but this will at least find host with them in the resouce tracker | |
| 15:26:25 | sean-k-mooney | so ya in the long run i agree i think this better and also that yes this is a new thing and setting precident | |
| 15:26:46 | gibi | awesome, then we are on the same page | |
| 15:27:24 | sean-k-mooney | am since we are talking about pci i agree with you regfarding the pack/spread change. i raised the behaivor delta in a prevous version | |
| 15:28:06 | gibi | ohh, cool then | |
| 15:28:21 | sean-k-mooney | i kind fo feel like we shoudl ignore the pack config option for pci devices for backward compatiablity with the excption that if you ask for one we shoudl actully prefer the nodes with pci devices | |