| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-14 | |||
| 10:10:18 | gibi | bauzas: anyhow, i will know more in 2 hours when i finally download and run the test | |
| 10:10:39 | bauzas | gibi: so, I tried to look at where we were, and I think we have a race | |
| 10:10:57 | stephenfin | ioni: You might want to look at commits d983234288728427235ef2c1f355ec135119b865 and cab8139498c7ea6b05cfdc8b4997276051b943fc in nova | |
| 10:11:15 | stephenfin | The first of those probably should have been backported, if it wasn't | |
| 10:11:45 | stephenfin | Looks like it is in stable/queens, actually | |
| 10:11:53 | bauzas | gibi: when https://review.openstack.org/#/c/631559/5/nova/tests/functional/libvirt/test_reshape.py@123, we don't have yet VGPU inventory since we just changed the conf opt | |
| 10:12:38 | bauzas | gibi: so, when entering .start(), we eventually go into https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L942 | |
| 10:12:44 | stephenfin | ioni: OK, so override the default and set it to '' (the empty string) | |
| 10:13:09 | ioni | stephenfin, ok, testing | |
| 10:13:20 | bauzas | gibi: but the provider tree we pass is not already having VGPUs, hence the reshape being not triggered | |
| 10:13:23 | ioni | i think i did test that but i just found out that i need to restart the instance | |
| 10:13:31 | bauzas | gibi: which is normal | |
| 10:13:34 | ioni | to start with the right options in libvirt | |
| 10:13:41 | bauzas | gibi: now, I think I see the problem actually | |
| 10:13:58 | bauzas | gibi: we should directly return inventories with VGPU on the child | |
| 10:14:14 | bauzas | gibi: but I made it conditional | |
| 10:14:28 | bauzas | gibi: anyway, continuing to dig | |
| 10:14:41 | bauzas | I feel I made a mistake and I'm close to find it | |
| 10:14:49 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Avoid redundant initialize_connection on source post live migration https://review.openstack.org/636895 | |
| 10:17:34 | openstackgerrit | sahid proposed openstack/nova master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610694 | |
| 10:18:05 | stephenfin | ioni: Right. You might even need to rebuild, actually. I'm not sure | |
| 10:18:14 | stephenfin | ioni: Just boot an new instance | |
| 10:18:17 | stephenfin | *a | |
| 10:21:59 | openstackgerrit | sahid proposed openstack/os-traits master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610065 | |
| 10:23:42 | ioni | stephenfin, thanks. it works | |
| 10:24:12 | stephenfin | ioni: \o/ I would suggest filing a bug against openstack-ansible to get them to override that default out of the box | |
| 10:24:41 | ioni | stephenfin, with default, i ment that nova starts with that default | |
| 10:24:52 | ioni | default nothing is confgured in nova.conf related to keymap | |
| 10:25:07 | stephenfin | ioni: We can't do it upstream because that would require everyone use noVNC 1.0. OSA can because they're deciding what version of that package gets installed | |
| 10:25:19 | stephenfin | ioni: Correct. I'm suggesting OSA override the nova default | |
| 10:25:27 | ioni | stephenfin, alright | |
| 10:28:00 | lyarwood | mdbooth: ack thanks, would you mind throwing a +1 on there to indicate that please? | |
| 10:40:27 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: tox: Don't write byte code (maybe) https://review.openstack.org/636917 | |
| 10:46:40 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/queens: tox: Don't write byte code (maybe) https://review.openstack.org/636918 | |
| 10:47:12 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/pike: tox: Don't write byte code (maybe) https://review.openstack.org/636919 | |
| 10:51:12 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/ocata: tox: Don't write byte code (maybe) https://review.openstack.org/636921 | |
| 10:53:29 | stephenfin | lyarwood: There's a couple of super easy, totally trivial backports there for you, if you fancy getting some stat padding in ^^^ :D | |
| 10:56:27 | stephenfin | sean-k-mooney: Thoughts on mriedem's comments here? https://review.openstack.org/#/c/635170/2/nova/virt/libvirt/vif.py | |
| 11:10:08 | lyarwood | stephenfin: ack | |
| 11:44:05 | maciejjozefczyk | sean-k-mooney: Hey, if you're interested I can provide you cherry-picks to Newton of patches about cache, it works well on our production | |
| 11:53:52 | openstackgerrit | Stephen Finucane proposed openstack/os-vif master: Add 'SUPPORT_BW_CONFIG' option to VIFs https://review.openstack.org/636933 | |
| 12:17:30 | gibi | bauzas: If I understand correctly the functional test creates VGPU inventory on the root, and a VM even boots with a VGPU before the reshape | |
| 12:18:13 | gibi | bauzas: I saw in the debug log of the test run that the VM has an mdev in the xml | |
| 12:18:16 | gibi | <hostdev mode="subsystem" type="mdev" managed="no" model="vfio-pci"> | |
| 12:18:19 | gibi | <source> | |
| 12:18:22 | gibi | <address uuid="4b20d080-1b54-4048-85b3-a6a62d165c01"/> | |
| 12:18:24 | gibi | </source> | |
| 12:18:27 | gibi | </hostdev> | |
| 12:19:00 | gibi | bauzas: I guess that the problem can be that the VGPU inventory is not added by the compute itself to the root RP but added by the functional test manually. | |
| 12:23:59 | gibi | bauzas: somehow if not the compute adds the VGPU to the root RP then the compute does not see it later when checking if reshape is needed. | |
| 12:25:03 | gibi | bauzas: it seems like the provider_tree cache is not updated after the VGPU is added to the root RP by the functional test, even if the functional test simulated the compute restart | |
| 12:25:43 | gibi | bauzas: I will dig into this. It can be an artifact of the way the functional test a) adds the VGPU inventory to the root RP or b) the way the tests simulates the compute service restart | |
| 12:33:23 | alex_xu | stephenfin: thanks for take care https://blueprints.launchpad.net/nova/+spec/remove-force-flag-from-live-migrate-and-evacuate | |
| 12:48:37 | openstackgerrit | Andrey Volkov proposed openstack/nova master: AZ list performance optimiztion: avoid double service list DB fetch https://review.openstack.org/636947 | |
| 12:50:06 | gibi | bauzas: yepp, it seems that when the func test simulates the compute restart the provider tree kept in the the report client is not dropped and therefore during the restart it is not refreshed | |
| 12:50:18 | gibi | bauzas: does the reportclient went to become a singleton? | |
| 12:58:12 | gibi | bauzas: nvm | |
| 12:58:17 | gibi | bauzas: it doesn't | |
| 13:02:15 | openstackgerrit | Andrey Volkov proposed openstack/nova master: AZ list performance optimization: avoid double service list DB fetch https://review.openstack.org/636947 | |
| 13:07:07 | gibi | bauzas: I managed to find the root case | |
| 13:07:14 | gibi | bauzas: it is in the test env | |
| 13:08:01 | gibi | bauzas: the service.stop service.start in the functional test does not properly simulate a real service restart, a lot of state are kept, like the resource tracker and the reportclient provider_tree cache. | |
| 13:08:25 | gibi | bauzas: I manually dropped the provider_tree cache between stop and start and now the test passes | |
| 13:10:18 | gibi | bauzas: pushing the fix to the test soon | |
| 13:11:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add functional test for libvirt vgpu reshape https://review.openstack.org/631559 | |
| 13:11:27 | gibi | bauzas: ^^ | |
| 13:18:19 | bauzas | gibi: yeah, I was about telling you that I didn't had a problem with the reshape when testing it in some hardware | |
| 13:19:19 | bauzas | gibi: see http://paste.openstack.org/show/745093/ | |
| 13:21:15 | openstackgerrit | Silvan Kaiser proposed openstack/nova master: Added mount fstype based validation of Quobyte mounts https://review.openstack.org/522245 | |
| 13:21:18 | bauzas | gibi: I just removed the -2 | |
| 13:29:55 | gibi | bauzas: the whole test problem boils down to the fact that it is not possible to create an old compute RP inventory directly from the compute code as that code was replaced by the new vgpu code. So the test should simulate that there is an old RP inventory with VGPU on the root | |
| 13:36:33 | bauzas | gibi: if you wanna test the real reshape, I agree | |
| 13:37:02 | bauzas | gibi: what does the test atm is making sure that on greenfields, you start with VGPU resources to be nested | |
| 13:37:04 | kaisers | stephenfin: Hi! FYI, i just ran an update to https://review.openstack.org/#/c/522245 based on yesterdays feedback :) | |
| 13:37:13 | mdbooth | Anybody got experience configuring devstack to use an https endpoint with a self-signed cert? | |
| 13:38:14 | mdbooth | To the best of my knowledge the cert is correctly installed in the system-wide ca-bundle, and certainly curl will now connect without --insecure. | |
| 13:38:19 | mdbooth | But openstackcli will not. | |
| 13:38:25 | mdbooth | And neither will Nova. | |
| 13:40:06 | mdbooth | '$ openstack --insecure volume list' *does* work, but afaik there's no similar option to disable nova's certificate validation when using cinderclient. | |
| 13:41:33 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Ignore VolumeAttachmentNotFound exception in compute.manager https://review.openstack.org/636954 | |
| 13:41:56 | mdbooth | But... is there some alternate ca-bundle somewhere? | |
| 13:42:07 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Provide a useful error message when trying to update non-compute services https://review.openstack.org/636955 | |
| 13:45:55 | bauzas | gibi: I'm not super happy with https://review.openstack.org/#/c/631559/5..6/nova/tests/functional/libvirt/test_reshape.py | |
| 13:48:53 | stephenfin | kaisers: Still got comments, unfortunately https://review.openstack.org/#/c/522245 | |
| 13:49:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change sqlalchemy warnings filter to an error https://review.openstack.org/636957 | |
| 13:55:17 | openstackgerrit | Mark Goddard proposed openstack/nova stable/queens: Don't emit warning when ironic properties are zero https://review.openstack.org/608611 | |
| 14:04:29 | mdbooth | Hmm, this is a difference in behaviour between urllib.requests (urllib2) and requests, which appears to use urllib3 | |
| 14:05:13 | mdbooth | The former appears to use the system ca bundle, the latter does not | |
| 14:06:37 | gibi | efried: I answered your questions / comments in https://review.openstack.org/#/c/616239/24 | |
| 14:07:16 | mdbooth | certifi, j'accuse | |
| 14:07:23 | gibi | bauzas: how can I make you happier about that reshape functional test? | |
| 14:08:25 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: Handle unicode characters in migration params https://review.openstack.org/636678 | |
| 14:10:34 | bauzas | gibi: self.compute is generated by nova.test.py which returns a ServiceFixture | |
| 14:11:23 | bauzas | gibi: if we really want to mock a real compute stop, we need to update the fixture | |
| 14:12:06 | bauzas | gibi: https://github.com/openstack/nova/blob/master/nova/test.py#L399 | |
| 14:13:06 | bauzas | gibi: somehow, you say in your comment that https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L83 doesn't work correctly | |
| 14:13:13 | bauzas | because we still have a cache | |
| 14:13:27 | bauzas | so we need to invalidate the cache there ^ | |
| 14:14:02 | gibi | bauzas: yeah, it would be nice to improve the fixture to support a real service restart | |
| 14:14:08 | gibi | bauzas: let me dig a bit about that | |