| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-01 | |||
| 12:05:28 | sean-k-mooney[m] | gibi at some point i stongly feel this needs to be fixed by default for everyone that uses nova and placment since the same broken behavior affects both | |
| 12:06:03 | sean-k-mooney[m] | so if we force a microverion for this we need to look at starting the process of raising or min micorversion | |
| 12:09:55 | gibi | sean-k-mooney[m]: ahh I see, so the middleware would be optional therefore does not need a microversion. Yeah that could be tried. Probably on can say that becomes a config driven API but paste.ini is created exactly for that kind of things | |
| 12:10:37 | sean-k-mooney1 | ya i was lookign are our exising middelware it does not look like it would be that hard to create one that did this | |
| 12:11:14 | sean-k-mooney1 | basically using https://github.com/openstack/nova/blob/master/nova/api/openstack/identity.py as a template | |
| 12:12:07 | sean-k-mooney | we would then add a new filter lin and add that filter to the pipeline https://github.com/openstack/nova/blob/master/etc/nova/api-paste.ini#L68-L81 | |
| 12:12:53 | sean-k-mooney | i belive that is how that work but i have never had a need to do that before but that seam to be the patteren | |
| 12:14:24 | sean-k-mooney | i read over most of our docs last night to try and find if they provdied any guidence on if repated arges were ever intended to be support and i cant fine any to that efffect for what its worth | |
| 12:16:43 | sean-k-mooney | with that said the https://github.com/openstack/nova/blob/master/doc/source/reference/stable-api.rst#v2-api-compatibility-mode-based-on-v21-api doc does imply that addtion request parmaters are not intended to be allowed | |
| 12:16:57 | sean-k-mooney | """v2.1 API is exactly same as v2 API except strong input validation with no additional request parameter allowed and Microversion feature.""" | |
| 12:21:16 | gibi | "additional" probably means "unknown to nova" not "repeated know" params | |
| 12:21:38 | gibi | as we allowed (and ignored) unknown params in the past | |
| 12:22:51 | gibi | but I agree, I think we need to put a rule to repeated params | |
| 12:27:25 | pmonteir | Good morning everybody! Does anybody know how the "live_migration_downtime" parameter was tested? Been having some trouble trying to understand how this works | |
| 12:29:21 | sean-k-mooney | pmonteir: its not really. we just pass that to libvirt nova is not in contol of the downtime | |
| 12:29:43 | sean-k-mooney | once nova start the migration libvirt is basically in charge until its done | |
| 12:30:17 | sean-k-mooney | we can call libvirt with addtional commands like force complete or abort | |
| 12:31:11 | sean-k-mooney | but nova does not activly monitor the downtime and progress we passivly recive the events form libvirt and we can query it in resonce to an api request but we just get out of libvirts way and let it do its thing | |
| 12:34:15 | pmonteir | Oh... I see, so if the max downtime is hit, that's just something we let libvirt handle? | |
| 12:35:33 | sean-k-mooney | i can check, in the case of max im not sure but the normal pauses ectra are not done by nova | |
| 12:36:53 | sean-k-mooney | we configure the max down time on the guest https://github.com/openstack/nova/blob/master/nova/virt/libvirt/migration.py#L475-L527 | |
| 12:37:17 | sean-k-mooney | im just checkign to see if we handel when its exceeded in nova or preconfigure the action to take | |
| 12:38:25 | pmonteir | Yh, that's exactly the thing I'm having some trouble trying to find out. If something is actually done when its exceeded | |
| 12:39:23 | sean-k-mooney | i know we have action if the over all migration timeout is exceeded but not sure about max_downtime | |
| 12:39:32 | sean-k-mooney | kashyap: do you happen to know ^ | |
| 12:41:01 | kashyap | I don't remember off-hand, have to dig in too | |
| 12:42:00 | sean-k-mooney | pmonteir: what are you trying to achive by modifyign this by the way | |
| 12:42:10 | sean-k-mooney | pmonteir: i assume you have seen https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/doc/source/admin/configuring-migrations.rst#advanced-configuration-for-kvm-and-qemu | |
| 12:42:45 | kashyap | Yeah, let's step back to understand the bigger picture | |
| 12:43:15 | sean-k-mooney | the auto convergence and post copy options might be of interest to you if you are seeign excessive downtime | |
| 12:43:47 | sean-k-mooney | if you are seeing network downtime/ping loss that is likely unrealted to this | |
| 12:44:23 | pmonteir | Yh, I have. So basically I was trying to understand this parameter and test it out. But I wasn't able to see it in action (the max downtime value being exceeded and a timeout being triggered) | |
| 12:45:09 | sean-k-mooney | unless the vm is hevially loaded and dirtying memory you wont hit this in a normal migration | |
| 12:47:03 | pmonteir | I was trying to migrate whilst dirtying memory, the migration kept going for some time and (once) I used "virsh list" and the vm being migrated got paused and stayed like that for a while, which I found it weird | |
| 12:47:50 | sean-k-mooney | pmonteir: the migration timeout proably got exceeded and use force-compelte | |
| 12:48:14 | sean-k-mooney | https://github.com/openstack/nova/blob/0d84833e9688e0df97f3d24e06025e512bca3ce3/nova/conf/libvirt.py#L372-L388 | |
| 12:48:21 | sean-k-mooney | its our default | |
| 12:49:12 | sean-k-mooney | that is based on live_migration_completion_timeout not based on downtime and downtime has no effect on how long the vm will be paused | |
| 12:50:22 | sean-k-mooney | pmonteir: if that is what happened you should see https://github.com/openstack/nova/blob/2f644a82fec13bad8fcdfa195c9316a6f09ee15a/nova/virt/libvirt/migration.py#L468-L469 in the log | |
| 12:51:30 | kashyap | pmonteir: So, one more way to make sure libvirt is actually *setting* the thing by passing it to QEMU is to observe the libvirt/QEMU interaction logs | |
| 12:51:34 | kashyap | pmonteir: You can do it this way: | |
| 12:53:09 | kashyap | (1) Config the libvirt/QEMU log filters using the 'virt-admin' tool (it gets installed as part of "libvirt-daemon" package on Fedora; check for your distro): | |
| 12:53:14 | kashyap | $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log" | |
| 12:53:17 | kashyap | $> virt-admin daemon-log-filters "1:qemu_monitor" | |
| 12:53:59 | kashyap | (2) Migrate your guest from source to destination (and also make sure you've set the "_downtime" config parameter) | |
| 12:54:29 | kashyap | (3) That's it. Now you can `grep` for "downtime-limit" on source and destination libvirtd.log | |
| 12:54:45 | kashyap | Or ... you can post the output of this: grep -Ei '(Send Command|QEMU_MONITOR_RECV_)' /var/log/libvirt/libvirtd.log | |
| 12:55:04 | kashyap | In a paste-bin, I can analyze it. (Get the above output from both source and destination compute nodes) | |
| 12:55:22 | sean-k-mooney | gibi: im more or less +2+w on https://review.opendev.org/c/openstack/nova/+/792356/8//COMMIT_MSG but do you want me to hold off so that teh implemented ... can be added? | |
| 12:57:05 | gibi | sean-k-mooney: no, lets get it landed | |
| 12:57:49 | pmonteir | I will try config those logs and see what I can get. Thank you guys! | |
| 12:58:18 | sean-k-mooney | gibi: ok ill hit it when i get back just going to grab a drink | |
| 12:58:24 | gibi | ack | |
| 12:58:29 | dmitriis | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/826675/5/check/nova-next/3661dcb/testr_results.html | |
| 12:58:29 | dmitriis | sean-k-mooney: do you know if test_tagged_attachment is known to be flaky? Seem to be hitting it intermittently in both https://review.opendev.org/c/openstack/nova/+/819494 and https://review.opendev.org/c/openstack/nova/+/826675 | |
| 13:04:57 | kashyap | pmonteir: Also, a broader tip: whenever you're debugging something that involves libvirt and QEMU: the above logs with the said filters are useful to investigate. (Upstream DevStack actually logs these by default) | |
| 13:06:28 | sean-k-mooney | dmitriis: not that im aware of but volume attachment can be | |
| 13:06:42 | pmonteir | kashyap: Oh, thanks! I'm kinda noob to openstack/libvirt so this is really helpful! :D | |
| 13:07:20 | kashyap | pmonteir: No prob. The above logs are specifically good to debug all live migration issues | |
| 13:07:51 | pmonteir | got it! | |
| 13:08:56 | kashyap | pmonteir: Oh, wait. I had an error in the second step. It needs a bit more: | |
| 13:10:16 | kashyap | Not the second step, but the second command in the first step, for filters. The correct set of filters are: | |
| 13:10:16 | dmitriis | sean-k-mooney: ack, ty | |
| 13:10:20 | kashyap | $> virt-admin daemon-log-filters "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | |
| 13:10:37 | kashyap | pmonteir: --^ Make the above edit, if you took a note of this. | |
| 13:12:06 | pmonteir | oh, nice, thanks! | |
| 14:47:23 | gibi | bauzas: fyi I've filed a gate-failure bug https://bugs.launchpad.net/nova/+bug/1959677 it is happening daily | |
| 14:47:38 | bauzas | thanks, I was about to update the agenda | |
| 14:58:17 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833 | |
| 14:58:17 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | |
| 14:58:18 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Bump os-traits to 2.7.0 https://review.opendev.org/c/openstack/nova/+/826675 | |
| 14:58:18 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Introduce remote_managed tag for PCI devs https://review.opendev.org/c/openstack/nova/+/824834 | |
| 14:58:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | |
| 14:58:19 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | |
| 15:04:30 | dmitriis | sean-k-mooney, gibi: ta for the reviews. Fixed the first two, looking at gibi's comments to the third one. | |
| 15:05:24 | gibi | dmitriis: ack, I will check back at some point | |
| 15:15:48 | gibi | sean-k-mooney: when you read further into the smartnic series, I would like to see your opinion about the dynamicy capability handling at https://review.opendev.org/c/openstack/nova/+/812111/16/nova/compute/resource_tracker.py#1176 I have not problem with it but it is someting new | |
| 15:16:47 | sean-k-mooney | dynamicy capability handeling? | |
| 15:17:06 | opendevreview | Jonathan Race proposed openstack/nova master: Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | |
| 15:17:21 | artom | TypeError: can only concatenate str (not "int") to str" a known thing | |
| 15:17:21 | artom | Is "'name': old_flavor['name'] + 'extra-' + data_utils.rand_int_id(), | |
| 15:17:21 | sean-k-mooney | oh the traits reporting | |
| 15:17:27 | artom | Happening in nova-next | |
| 15:17:51 | sean-k-mooney | gibi: we have something simiarl i belive else where but slightly less clever in implemenation | |
| 15:17:56 | artom | Looks like a Tempest code error, perhaps data_utils.rand_int_id() changed to return an int or something, but I feel like if it was tempest it would be all over the place | |
| 15:18:24 | gibi | artom: that logic is a recent addition in tempest | |
| 15:18:52 | sean-k-mooney | artom: so ya that looks like it shoudl have a str() call | |
| 15:18:59 | artom | gibi, ah, right, cae966812a4a5070c3e7f82d16ebe697da57e5c5 | |
| 15:19:04 | artom | Well, it's breaking nova-next | |
| 15:19:04 | sean-k-mooney | althoguh personally i woudl be tempted to use an fstring | |
| 15:19:12 | gibi | yepp you were faster finding it | |
| 15:19:23 | gibi | artom: thanks | |
| 15:20:00 | sean-k-mooney | the fix is firly simple just use sting interpulation of fstring instead fo concationation or cast | |
| 15:20:01 | gibi | sean-k-mooney: cool | |
| 15:20:58 | sean-k-mooney | normally we do simple things like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L395-L420 | |
| 15:20:59 | gibi | artom: the orginal tempest fix did not trigger nova-next (or an ovs full tempest from neutron) and therefore the test case that is changed was not run for the tempest patch | |
| 15:21:19 | artom | Yeah, all good. Quick enough fix | |
| 15:21:30 | artom | It'll take me longer to file the LP bug than changing the code :P | |
| 15:21:41 | sean-k-mooney | but we also build trait dymically like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8412-L8434 | |
| 15:21:52 | gibi | artom: please add a DNM patch in nova that depends on the tempest fix to see nova-next is green after it :) | |