Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-01
12:53:09 kashyap (1) Config the libvirt/QEMU log filters using the 'virt-admin' tool (it gets installed as part of "libvirt-daemon" package on Fedora; check for your distro):
12:53:14 kashyap $> virt-admin daemon-log-outputs "1:file:/var/log/libvirt/libvirtd.log"
12:53:17 kashyap $> virt-admin daemon-log-filters "1:qemu_monitor"
12:53:59 kashyap (2) Migrate your guest from source to destination (and also make sure you've set the "_downtime" config parameter)
12:54:29 kashyap (3) That's it. Now you can `grep` for "downtime-limit" on source and destination libvirtd.log
12:54:45 kashyap Or ... you can post the output of this: grep -Ei '(Send Command|QEMU_MONITOR_RECV_)' /var/log/libvirt/libvirtd.log
12:55:04 kashyap In a paste-bin, I can analyze it. (Get the above output from both source and destination compute nodes)
12:55:22 sean-k-mooney gibi: im more or less +2+w on https://review.opendev.org/c/openstack/nova/+/792356/8//COMMIT_MSG but do you want me to hold off so that teh implemented ... can be added?
12:57:05 gibi sean-k-mooney: no, lets get it landed
12:57:49 pmonteir I will try config those logs and see what I can get. Thank you guys!
12:58:18 sean-k-mooney gibi: ok ill hit it when i get back just going to grab a drink
12:58:24 gibi ack
12:58:29 dmitriis sean-k-mooney: do you know if test_tagged_attachment is known to be flaky? Seem to be hitting it intermittently in both https://review.opendev.org/c/openstack/nova/+/819494 and https://review.opendev.org/c/openstack/nova/+/826675
12:58:29 dmitriis https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/826675/5/check/nova-next/3661dcb/testr_results.html
13:04:57 kashyap pmonteir: Also, a broader tip: whenever you're debugging something that involves libvirt and QEMU: the above logs with the said filters are useful to investigate. (Upstream DevStack actually logs these by default)
13:06:28 sean-k-mooney dmitriis: not that im aware of but volume attachment can be
13:06:42 pmonteir kashyap: Oh, thanks! I'm kinda noob to openstack/libvirt so this is really helpful! :D
13:07:20 kashyap pmonteir: No prob. The above logs are specifically good to debug all live migration issues
13:07:51 pmonteir got it!
13:08:56 kashyap pmonteir: Oh, wait. I had an error in the second step. It needs a bit more:
13:10:16 dmitriis sean-k-mooney: ack, ty
13:10:16 kashyap Not the second step, but the second command in the first step, for filters. The correct set of filters are:
13:10:20 kashyap $> virt-admin daemon-log-filters "1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util"
13:10:37 kashyap pmonteir: --^ Make the above edit, if you took a note of this.
13:12:06 pmonteir oh, nice, thanks!
14:47:23 gibi bauzas: fyi I've filed a gate-failure bug https://bugs.launchpad.net/nova/+bug/1959677 it is happening daily
14:47:38 bauzas thanks, I was about to update the agenda
14:58:17 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199
14:58:17 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833
14:58:18 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Introduce remote_managed tag for PCI devs https://review.opendev.org/c/openstack/nova/+/824834
14:58:18 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: Bump os-traits to 2.7.0 https://review.opendev.org/c/openstack/nova/+/826675
14:58:19 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835
14:58:19 opendevreview Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111
15:04:30 dmitriis sean-k-mooney, gibi: ta for the reviews. Fixed the first two, looking at gibi's comments to the third one.
15:05:24 gibi dmitriis: ack, I will check back at some point
15:15:48 gibi sean-k-mooney: when you read further into the smartnic series, I would like to see your opinion about the dynamicy capability handling at https://review.opendev.org/c/openstack/nova/+/812111/16/nova/compute/resource_tracker.py#1176 I have not problem with it but it is someting new
15:16:47 sean-k-mooney dynamicy capability handeling?
15:17:06 opendevreview Jonathan Race proposed openstack/nova master: Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
15:17:21 sean-k-mooney oh the traits reporting
15:17:21 artom Is "'name': old_flavor['name'] + 'extra-' + data_utils.rand_int_id(),
15:17:21 artom TypeError: can only concatenate str (not "int") to str" a known thing
15:17:27 artom Happening in nova-next
15:17:51 sean-k-mooney gibi: we have something simiarl i belive else where but slightly less clever in implemenation
15:17:56 artom Looks like a Tempest code error, perhaps data_utils.rand_int_id() changed to return an int or something, but I feel like if it was tempest it would be all over the place
15:18:24 gibi artom: that logic is a recent addition in tempest
15:18:52 sean-k-mooney artom: so ya that looks like it shoudl have a str() call
15:18:59 artom gibi, ah, right, cae966812a4a5070c3e7f82d16ebe697da57e5c5
15:19:04 sean-k-mooney althoguh personally i woudl be tempted to use an fstring
15:19:04 artom Well, it's breaking nova-next
15:19:12 gibi yepp you were faster finding it
15:19:23 gibi artom: thanks
15:20:00 sean-k-mooney the fix is firly simple just use sting interpulation of fstring instead fo concationation or cast
15:20:01 gibi sean-k-mooney: cool
15:20:58 sean-k-mooney normally we do simple things like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L395-L420
15:20:59 gibi artom: the orginal tempest fix did not trigger nova-next (or an ovs full tempest from neutron) and therefore the test case that is changed was not run for the tempest patch
15:21:19 artom Yeah, all good. Quick enough fix
15:21:30 artom It'll take me longer to file the LP bug than changing the code :P
15:21:41 sean-k-mooney but we also build trait dymically like this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8412-L8434
15:21:52 gibi artom: please add a DNM patch in nova that depends on the tempest fix to see nova-next is green after it :)
15:21:58 sean-k-mooney gibi: is your concern that htis is a mor dynimic capblity
15:22:03 sean-k-mooney not just a dynmic trait
15:22:04 artom gibi, good idea
15:22:15 gibi sean-k-mooney: this is a new thing and it will create precedent
15:22:32 gibi sean-k-mooney: I mean that we dinamically apply capability based on resource inventory
15:23:19 gibi sean-k-mooney: so I'd like to make a conscious introducing it
15:23:21 sean-k-mooney oh as in it only reports the trait when we have a pool that support it
15:23:26 gibi yepp
15:23:41 sean-k-mooney right
15:23:50 gibi but personally I have no problem with it
15:24:01 sean-k-mooney well we technially dont need to do that actully
15:24:05 sean-k-mooney i dont really either
15:24:16 sean-k-mooney but we coudl jsut always report it
15:24:20 sean-k-mooney if the host support it
15:24:34 gibi it is basically indirectly config driver (by the whitelist config) and we already have config driven capabilities
15:24:37 sean-k-mooney the inventory check is going to be enforce by the pci filter after the fact
15:24:48 gibi s/driver/driven/
15:24:50 sean-k-mooney yes we do
15:25:07 sean-k-mooney the main advantage to having it is the filtering will partly be done by placment
15:25:44 gibi yepp
15:25:47 gibi that is a good thing
15:25:59 sean-k-mooney the smart nics port dont have inventories in placement currntly but this will at least find host with them in the resouce tracker
15:26:25 sean-k-mooney so ya in the long run i agree i think this better and also that yes this is a new thing and setting precident
15:26:46 gibi awesome, then we are on the same page
15:27:24 sean-k-mooney am since we are talking about pci i agree with you regfarding the pack/spread change. i raised the behaivor delta in a prevous version
15:28:06 gibi ohh, cool then
15:28:21 sean-k-mooney i kind fo feel like we shoudl ignore the pack config option for pci devices for backward compatiablity with the excption that if you ask for one we shoudl actully prefer the nodes with pci devices
15:28:37 sean-k-mooney today we dont prefer if you ask for a pci device we jsut avoid if we dont
15:29:13 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: Test fix for bug 1959682 https://review.opendev.org/c/openstack/nova/+/827306
15:29:15 sean-k-mooney i had said in the previous version i was ok with debating that in a followup if others agreed to mvoe it forward
15:29:30 sean-k-mooney i have remvoed my +2 for now so we can disucss it in the patch
15:31:03 gibi ok
15:46:46 opendevreview Jonathan Race proposed openstack/nova master: Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
15:50:25 bauzas reminder : nova meeting here in 10 mins
15:57:05 bauzas elodilles: no agenda for stable branches ?
15:59:40 elodilles bauzas: well, i forgot to update it well before you so i waited a bit... maybe for too long o:) sorry :)
15:59:47 bauzas elodilles: just do it now
15:59:49 bauzas I'll refresh
15:59:52 elodilles let me update it quickly :)
15:59:59 elodilles (not so much to update)
15:59:59 bauzas #startmeeting nova

Earlier   Later