Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-02-14
09:46:32 bauzas gibi: around ?
09:48:41 bauzas gibi: I don't get why when entering the reshape here https://review.openstack.org/#/c/599208/13/nova/virt/libvirt/driver.py@7000 we don't have the VGPU inventory
09:48:53 gibi bauzas: just from a phone. I will be around properly during my afternoon (UTC+1)
09:49:13 bauzas gibi: ack, I probably need your help to understand the problem
09:49:50 bauzas gibi: just to explain the issue, we don't trigger the reshape since the root inventory we have at first time doesn't have yet the VGPU RC
09:49:57 bauzas it will be there later
09:50:54 gibi good hint, can be some race condition in the test
09:52:47 bauzas gibi: I feel we probably trigger the reshape *before* we add the inventory to the root RP
10:02:13 ioni hello
10:02:33 ioni did you guys noticed that novnc sometimes doesn't have the right keymap, i think. the problem is that i cannot type | in the console, whenever i try to use shift+\ for |, the character is > instead of |
10:02:58 openstackgerrit Chris Dent proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972
10:05:17 lyarwood mdbooth_: https://review.openstack.org/#/c/551349/ - Anything more you wanted to add to this btw?
10:05:32 stephenfin ioni: What version of noVNC are you using? You need 1.0 for proper keymap support
10:06:01 ioni stephenfin, it's the one that comes with openstack ansible queens version
10:06:32 ioni novncproxy_git_repo: https://github.com/novnc/noVNC
10:06:32 ioni novncproxy_git_install_branch: 37b4d13db81e0e80e117c07b86ff98714c7b6b1a # Frozen HEAD of "master" as of 10.02.2018
10:06:52 ioni stephenfin, i think on rocky is the same
10:07:31 ioni stephenfin, i tried latest master and had the same problem but it was giving me the wrong server console for some reason and i reverted
10:07:47 stephenfin ioni: Oh yeah, that's 1.0.0, which should be fine
10:07:58 ioni had the same problem with |
10:08:23 stephenfin Then I'm not sure. You should not have the '[vnc] keymap' nova.conf setting configured
10:08:32 stephenfin So if that's set, unset it
10:08:44 ioni ok so keymap = None ?
10:09:15 gibi bauzas: the test first boot an instance while the vgpu resource is still on the root RP
10:09:20 ioni default on queens is en-us
10:09:46 gibi bauzas: and check the allocation and usages
10:09:47 ioni 2019-02-14 12:09:31.814 2988 DEBUG oslo_service.service [req-b9b6b811-d6c2-49f7-9006-47e60b8d17a7 - - - - -] vnc.keymap = en-us log_opt_values /openstack/venvs/nova-17.1.7/lib/python2.7/site-packages/oslo_config/cfg.py:2898
10:09:47 ioni 2019-02-14 12:09:31.874 2988 DEBUG oslo_service.service [req-b9b6b811-d6c2-49f7-9006-47e60b8d17a7 - - - - -] spice.keymap = en-us log_opt_values /openstack/venvs/nova-17.1.7/lib/python2.7/site-packages/oslo_config/cfg.py:2898
10:09:58 mdbooth_ lyarwood: Not specifically.
10:10:00 ioni it's starts with this default if is not set
10:10:18 gibi bauzas: anyhow, i will know more in 2 hours when i finally download and run the test
10:10:39 bauzas gibi: so, I tried to look at where we were, and I think we have a race
10:10:57 stephenfin ioni: You might want to look at commits d983234288728427235ef2c1f355ec135119b865 and cab8139498c7ea6b05cfdc8b4997276051b943fc in nova
10:11:15 stephenfin The first of those probably should have been backported, if it wasn't
10:11:45 stephenfin Looks like it is in stable/queens, actually
10:11:53 bauzas gibi: when https://review.openstack.org/#/c/631559/5/nova/tests/functional/libvirt/test_reshape.py@123, we don't have yet VGPU inventory since we just changed the conf opt
10:12:38 bauzas gibi: so, when entering .start(), we eventually go into https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L942
10:12:44 stephenfin ioni: OK, so override the default and set it to '' (the empty string)
10:13:09 ioni stephenfin, ok, testing
10:13:20 bauzas gibi: but the provider tree we pass is not already having VGPUs, hence the reshape being not triggered
10:13:23 ioni i think i did test that but i just found out that i need to restart the instance
10:13:31 bauzas gibi: which is normal
10:13:34 ioni to start with the right options in libvirt
10:13:41 bauzas gibi: now, I think I see the problem actually
10:13:58 bauzas gibi: we should directly return inventories with VGPU on the child
10:14:14 bauzas gibi: but I made it conditional
10:14:28 bauzas gibi: anyway, continuing to dig
10:14:41 bauzas I feel I made a mistake and I'm close to find it
10:14:49 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: Avoid redundant initialize_connection on source post live migration https://review.openstack.org/636895
10:17:34 openstackgerrit sahid proposed openstack/nova master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610694
10:18:05 stephenfin ioni: Right. You might even need to rebuild, actually. I'm not sure
10:18:14 stephenfin ioni: Just boot an new instance
10:18:17 stephenfin *a
10:21:59 openstackgerrit sahid proposed openstack/os-traits master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610065
10:23:42 ioni stephenfin, thanks. it works
10:24:12 stephenfin ioni: \o/ I would suggest filing a bug against openstack-ansible to get them to override that default out of the box
10:24:41 ioni stephenfin, with default, i ment that nova starts with that default
10:24:52 ioni default nothing is confgured in nova.conf related to keymap
10:25:07 stephenfin ioni: We can't do it upstream because that would require everyone use noVNC 1.0. OSA can because they're deciding what version of that package gets installed
10:25:19 stephenfin ioni: Correct. I'm suggesting OSA override the nova default
10:25:27 ioni stephenfin, alright
10:28:00 lyarwood mdbooth: ack thanks, would you mind throwing a +1 on there to indicate that please?
10:40:27 openstackgerrit Stephen Finucane proposed openstack/nova stable/rocky: tox: Don't write byte code (maybe) https://review.openstack.org/636917
10:46:40 openstackgerrit Stephen Finucane proposed openstack/nova stable/queens: tox: Don't write byte code (maybe) https://review.openstack.org/636918
10:47:12 openstackgerrit Stephen Finucane proposed openstack/nova stable/pike: tox: Don't write byte code (maybe) https://review.openstack.org/636919
10:51:12 openstackgerrit Stephen Finucane proposed openstack/nova stable/ocata: tox: Don't write byte code (maybe) https://review.openstack.org/636921
10:53:29 stephenfin lyarwood: There's a couple of super easy, totally trivial backports there for you, if you fancy getting some stat padding in ^^^ :D
10:56:27 stephenfin sean-k-mooney: Thoughts on mriedem's comments here? https://review.openstack.org/#/c/635170/2/nova/virt/libvirt/vif.py
11:10:08 lyarwood stephenfin: ack
11:44:05 maciejjozefczyk sean-k-mooney: Hey, if you're interested I can provide you cherry-picks to Newton of patches about cache, it works well on our production
11:53:52 openstackgerrit Stephen Finucane proposed openstack/os-vif master: Add 'SUPPORT_BW_CONFIG' option to VIFs https://review.openstack.org/636933
12:17:30 gibi bauzas: If I understand correctly the functional test creates VGPU inventory on the root, and a VM even boots with a VGPU before the reshape
12:18:13 gibi bauzas: I saw in the debug log of the test run that the VM has an mdev in the xml
12:18:16 gibi <hostdev mode="subsystem" type="mdev" managed="no" model="vfio-pci">
12:18:19 gibi <source>
12:18:22 gibi <address uuid="4b20d080-1b54-4048-85b3-a6a62d165c01"/>
12:18:24 gibi </source>
12:18:27 gibi </hostdev>
12:19:00 gibi bauzas: I guess that the problem can be that the VGPU inventory is not added by the compute itself to the root RP but added by the functional test manually.
12:23:59 gibi bauzas: somehow if not the compute adds the VGPU to the root RP then the compute does not see it later when checking if reshape is needed.
12:25:03 gibi bauzas: it seems like the provider_tree cache is not updated after the VGPU is added to the root RP by the functional test, even if the functional test simulated the compute restart
12:25:43 gibi bauzas: I will dig into this. It can be an artifact of the way the functional test a) adds the VGPU inventory to the root RP or b) the way the tests simulates the compute service restart
12:33:23 alex_xu stephenfin: thanks for take care https://blueprints.launchpad.net/nova/+spec/remove-force-flag-from-live-migrate-and-evacuate
12:48:37 openstackgerrit Andrey Volkov proposed openstack/nova master: AZ list performance optimiztion: avoid double service list DB fetch https://review.openstack.org/636947
12:50:06 gibi bauzas: yepp, it seems that when the func test simulates the compute restart the provider tree kept in the the report client is not dropped and therefore during the restart it is not refreshed
12:50:18 gibi bauzas: does the reportclient went to become a singleton?
12:58:12 gibi bauzas: nvm
12:58:17 gibi bauzas: it doesn't
13:02:15 openstackgerrit Andrey Volkov proposed openstack/nova master: AZ list performance optimization: avoid double service list DB fetch https://review.openstack.org/636947
13:07:07 gibi bauzas: I managed to find the root case
13:07:14 gibi bauzas: it is in the test env
13:08:01 gibi bauzas: the service.stop service.start in the functional test does not properly simulate a real service restart, a lot of state are kept, like the resource tracker and the reportclient provider_tree cache.
13:08:25 gibi bauzas: I manually dropped the provider_tree cache between stop and start and now the test passes
13:10:18 gibi bauzas: pushing the fix to the test soon
13:11:21 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add functional test for libvirt vgpu reshape https://review.openstack.org/631559
13:11:27 gibi bauzas: ^^
13:18:19 bauzas gibi: yeah, I was about telling you that I didn't had a problem with the reshape when testing it in some hardware
13:19:19 bauzas gibi: see http://paste.openstack.org/show/745093/
13:21:15 openstackgerrit Silvan Kaiser proposed openstack/nova master: Added mount fstype based validation of Quobyte mounts https://review.openstack.org/522245
13:21:18 bauzas gibi: I just removed the -2

Earlier   Later