| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-07 | |||
| 09:55:41 | bauzas | gibi: I don't want to change this | |
| 09:55:58 | bauzas | gibi: I'm just saying there is a piece of metadata information that neutron has to know | |
| 09:56:04 | bauzas | which is carried by Nova | |
| 09:56:21 | bauzas | but which could be provided by Cyborg before the nova port creation call | |
| 09:56:26 | gibi | sure. I'm not suggesting to change this. I just noted down the logical problem about letting cyborg handle neutron | |
| 09:56:44 | gibi | bauzas: nova does not create the neutron port for sriov | |
| 09:56:58 | gibi | those ports need to be pre-created and provided in the boot request to nova | |
| 09:57:05 | gibi | afaik | |
| 09:57:10 | bauzas | oh you're right | |
| 09:57:15 | bauzas | we don't late bind | |
| 09:57:22 | bauzas | actually, we bind | |
| 09:57:26 | bauzas | but we don't create | |
| 10:05:26 | Yumeng | bauzas: emm IMHO, cyborg doesn't create the port in neutron during the boot. When Cyborg discovers SRIOV device and stores the device's info, user need to use Cyborg create a port/bandwith request, i.e. device_profile | |
| 10:05:56 | bauzas | yeah I was confused, my apologies | |
| 10:06:03 | bauzas | so, there is a pre-creation step | |
| 10:06:14 | bauzas | and then nova gets the port at boot time | |
| 10:06:25 | Yumeng | yes | |
| 10:10:03 | bauzas | OK, I think I still need to go thru the etherpad to understand the reasoning but that's a good first step | |
| 10:10:49 | Yumeng | gibi,bauzas: Thanks for your time! I am going off line now. my time is UTC+8. You can either ping me in IRC at #openstack-cyborg channel if you have more question. or just leave comments at ehterpad(topic 13):https://etherpad.opendev.org/p/cyborg-victoria-goals https://etherpad.opendev.org/p/cyborg-victoria-goals | |
| 10:11:04 | Yumeng | bauzas: thanks for taking time reading it! | |
| 10:11:10 | gibi | Yumeng: thanks I tried to add some not the nova etherpad. | |
| 10:11:17 | bauzas | gibi: fwiw, playing with stestr run --until-failure and then stestr --last helps me reproduce the failure everytime | |
| 10:11:52 | Yumeng | gibi: ok, thanks. I will go back to nova ehterpad later. | |
| 10:12:12 | gibi | bauzas: meantime I was able to reproduce the problem only runnig the single test case test_multiple_instance_create$ so it is not test case interference | |
| 10:12:35 | gibi | (it took 356 runs though) | |
| 10:12:51 | gibi | Yumeng: OK. have a nice evening | |
| 10:12:54 | bauzas | you mean, the single test method() without nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_multiple_instance_create_filling_up_capacity ? | |
| 10:13:02 | bauzas | Yumeng: ack, will do | |
| 10:13:23 | gibi | bauzas: yes a single test method nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_multiple_instance_create [7.793940s] ... FAILED | |
| 10:13:31 | bauzas | gibi: if so, the race is probably because we need to hold for a few things | |
| 10:14:05 | gibi | bauzas: yeah it is most likele a missing wait_for_<something> at some point in the test | |
| 10:14:07 | bauzas | gibi: lemme see if adding a pause of 1 sec will just solve the problem | |
| 10:14:18 | gibi | ack | |
| 10:14:24 | bauzas | gibi: because what's failing is that we lookup the mdevs | |
| 10:14:34 | bauzas | gibi: we should potentially wait a bit | |
| 10:15:02 | bauzas | this being said, we wait for the ACTIVE status | |
| 10:15:22 | bauzas | oh wait | |
| 10:15:30 | bauzas | this is a multiple instance creation | |
| 10:15:43 | bauzas | so we only check the first instance creation | |
| 10:16:05 | gibi | that could explain it | |
| 10:16:19 | bauzas | lemme try | |
| 10:16:38 | bauzas | --until-failure allows me to run the test easily | |
| 10:18:07 | bauzas | ahah, interesting | |
| 10:18:11 | bauzas | we don't hold | |
| 10:26:21 | bauzas | gibi: okay, changed my test to wait for all servers, running it | |
| 10:26:28 | gibi | cool | |
| 10:26:28 | bauzas | gibi: will leave it run for one hour or so | |
| 10:26:44 | gibi | if you push the patch up I can do the same while I have lunch | |
| 10:26:54 | bauzas | and if I no longer see any issues after, say, 200 runs, then we will know it's fixed | |
| 10:27:01 | bauzas | k | |
| 10:27:28 | bauzas | gibi: just writing a bug and uploading then | |
| 10:28:09 | gibi | ack | |
| 10:29:36 | bauzas | ah shit, they started it :p | |
| 10:30:44 | bauzas | https://bugs.launchpad.net/nova/+bug/1877281 | |
| 10:30:44 | openstack | Launchpad bug 1877281 in OpenStack Compute (nova) "vGPU multiple instance creation test is racey" [High,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 10:31:27 | bauzas | gibi: worth thinking about RC3 for this one ? :( | |
| 10:31:50 | bauzas | or, maybe we just release and then backport later | |
| 10:31:59 | bauzas | the issue isn't that bad | |
| 10:32:12 | jazeltq | hello gibi | |
| 10:32:41 | jazeltq | I have one question about nova-scheduler | |
| 10:35:56 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Wait for all servers to be active when testing vGPUs https://review.opendev.org/726087 | |
| 10:36:19 | bauzas | gibi: ^ | |
| 10:37:01 | bauzas | gibi: FYI (functional) [sbauza@sbauza nova]$ stestr --test-path=./nova/tests/functional run --until-failure nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_multiple_instance_create | |
| 10:42:21 | stephenfin | melwitt: When you're about, can you take a look at https://review.opendev.org/#/c/677444/ please? | |
| 10:48:39 | gibi | bauzas: I don't think we need an RC for a racy func test. It is OK to merge it after GA | |
| 12:00:36 | bauzas | gibi: okay, after 4560 seconds, no errors were found | |
| 12:00:57 | gibi | bauzas: no error seen in 412 runs | |
| 12:01:02 | gibi | so I agree you fixed it! | |
| 12:01:28 | gibi | nice work | |
| 12:27:54 | efried | stephenfin: go ahead and update that commit message. I'm here to re+1, for all the good it'll do ya. | |
| 12:35:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: config: Explicitly register 'remote_debug' CLI opts https://review.opendev.org/677444 | |
| 12:35:31 | bauzas | stephenfin: https://review.opendev.org/#/c/726087/1 closing https://bugs.launchpad.net/nova/+bug/1877281 | |
| 12:35:31 | openstack | Launchpad bug 1877281 in OpenStack Compute (nova) "vGPU multiple instance creation test is racey" [High,In progress] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 12:35:32 | stephenfin | efried: ack, thanks | |
| 12:35:39 | stephenfin | bauzas++ | |
| 12:37:55 | efried | stephenfin: done (with bonus) | |
| 12:39:48 | stephenfin | bauzas: I have an ask on https://review.opendev.org/#/c/726087/ | |
| 12:40:01 | stephenfin | if you don't want to, I can live with it as-is | |
| 12:43:59 | bauzas | stephenfin: no worries, I can write a new rev | |
| 12:44:06 | bauzas | there is no urgency to fix this race | |
| 12:44:18 | bauzas | we only get it for around 0.5% | |
| 12:45:55 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Wait for all servers to be active when testing vGPUs https://review.opendev.org/726087 | |
| 12:46:10 | bauzas | stephenfin: fer your plizure ^ | |
| 12:46:47 | stephenfin | +" | |
| 12:46:49 | stephenfin | *2 | |
| 13:18:57 | francoisp | bauzas, salut, when you have time: https://review.opendev.org/#/c/725272/ (this time for Rocky) - thanks | |
| 14:27:17 | bauzas | francoisp: ack, will try to do it asap | |
| 14:27:43 | francoisp | bauzas, thank you! | |
| 14:32:53 | openstackgerrit | Jiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD https://review.opendev.org/572805 | |
| 14:36:19 | openstackgerrit | Takashi Kajinami proposed openstack/nova master: Check py3 fix in python-barbicanclient https://review.opendev.org/726164 | |
| 14:57:54 | openstackgerrit | Sasha Andonov proposed openstack/nova master: rbd_utils: increase _destroy_volume timeout https://review.opendev.org/705764 | |
| 15:45:49 | gibi | nova meeting starts in 15 minutes on #openstack-meeting-3 | |
| 16:15:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Rewrite 'nova-manage db' docs to use proper directives https://review.opendev.org/677509 | |
| 16:15:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Use proper headers in nova-manage man page https://review.opendev.org/726191 | |
| 16:15:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Rewrite 'nova-manage api_db', 'cell_v2' docs https://review.opendev.org/726192 | |
| 16:15:02 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Rewrite 'nova-manage placement' docs https://review.opendev.org/726193 | |
| 16:15:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Only use section headers for HTML https://review.opendev.org/726194 | |
| 16:15:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document options for nova-*proxy https://review.opendev.org/726195 | |
| 16:44:23 | openstackgerrit | Shi Yan proposed openstack/nova master: Fix aggregate placement sync issue https://review.opendev.org/726201 | |
| 16:47:07 | openstackgerrit | Merged openstack/os-vif master: Remove six https://review.opendev.org/724966 | |