| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-07 | |||
| 08:27:47 | gibi | let me try running the test alon repeatedly | |
| 08:28:01 | bauzas | ok, because I only run the two multiple_create() methods | |
| 08:28:12 | bauzas | (looping over) | |
| 08:28:33 | bauzas | so, I wonder whether some global object would be raced | |
| 08:29:24 | gibi | yeah, that is possible | |
| 08:29:46 | gibi | I have to jump on a call now, but I will run the tests in different ways to see if it is an interference or not | |
| 08:30:12 | bauzas | kk thanks | |
| 08:33:17 | openstackgerrit | Jie Li proposed openstack/nova master: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/722187 | |
| 08:47:32 | bauzas | ah, bingo https://docs.openstack.org/os-testr/latest/user/subunit_trace.html | |
| 08:49:39 | bauzas | or rather https://stestr.readthedocs.io/en/latest/MANUAL.html#running-previously-failed-tests | |
| 08:50:03 | gibi | bauzas: running just the test_multiple_instance_create* test case I can still reproduce the problem | |
| 08:50:25 | gibi | but it needed like ~50 execution to get it once | |
| 08:51:24 | gibi | now trying just only the single test_multiple_instance_create$ case as I suspect that it is not an interference | |
| 08:52:19 | gibi | when I run the two test cases the, the test_multiple_instance_create$ failed, and the two case run in two separate executor so they cannot interfere | |
| 09:03:40 | bauzas | gibi: you can ask to use a single subunit | |
| 09:04:12 | gibi | bauzas: just paste me the command you want me to run and I will try that as well | |
| 09:04:12 | bauzas | gibi: that's why I'd like to get the subunit output from the failing job and trying to rerun it | |
| 09:04:47 | bauzas | gibi: https://stestr.readthedocs.io/en/latest/MANUAL.html#parallel-testing | |
| 09:04:48 | gibi | so far, after 77 execution no failure in the single test_multiple_instance_create$ test case execution | |
| 09:05:53 | bauzas | gibi: I got one after a shit number of executions | |
| 09:08:59 | Yumeng | Hi gibi! Good morning, I'm Yumeng from Cyborg Team. | |
| 09:09:20 | Yumeng | Cyborg wants to have a "cyborg/neutron/nova sriov integration" discussion with NOVA and NEUTRON during the Virtual PTG. | |
| 09:09:24 | Yumeng | Could you please find 1 hour for nova team to discuss this topic with cyborg and neutron team? | |
| 09:11:33 | Yumeng | I notice that there is a nova-neutron session on PTG Friday 13:00 UTC, can we join after that on Friday 15:00 UTC? | |
| 09:12:08 | gibi | Yumeng: hi! the schedule already closed so we have to use an exsiting booked slot for that discussion. Do you have a room booking from Friday 15:00 ? | |
| 09:13:42 | Yumeng | gibi: no, I don't have now. But I can email Kendall and book the room if it's ok for nova and neutron. | |
| 09:15:48 | gibi | Yumeng: I think we can try Friday 14:00 - 15:00 where we already have a room booked, or if you can get a room from Kendall then I we can do the discussion from 15:00 - 16:00 | |
| 09:17:19 | Yumeng | gibi: aha! Friday 14:00 - 15:00 is fantastic! I am just not sure if it is already occupied. | |
| 09:18:07 | gibi | Yumeng: I kept that as an overflow slot so I think we can use that for cyborg. If you need the neutron team present as well then you have to synch with slaweq too | |
| 09:19:21 | bauzas | Yumeng: tbh, what use case are you talking of ? | |
| 09:19:24 | Yumeng | Great! Thank you so much gibi! I will sync with slaweq! ^^ | |
| 09:19:33 | gibi | Yumeng: cool. I added it to the nova etherpad | |
| 09:19:41 | bauzas | SR-IOV but how ? | |
| 09:19:52 | Yumeng | bauzas: yes, exactly! | |
| 09:20:03 | gibi | Yumeng, bauzas: yeah that is my next request, please try to prepare some material before the discussion | |
| 09:20:19 | bauzas | Yumeng: okay, but why do you need all contributors ? | |
| 09:20:24 | bauzas | in nova and neutron ? | |
| 09:20:42 | bauzas | what kind of feature request would you want to do in Cyborg ? | |
| 09:20:53 | gibi | ^^ +1 | |
| 09:21:02 | Yumeng | We will try to write a spec to describe this topic. And one of our team member from Intel is preparing a POC. | |
| 09:21:45 | Yumeng | yes, a lot question should be answered. ^^ | |
| 09:22:34 | bauzas | Yumeng: before writing a spec, can you just summarize the reason ? | |
| 09:28:46 | bauzas | Yumeng: just saying, we still need to work on providing the NUMA topologies in Placement and then providing the PCI devices | |
| 09:29:25 | bauzas | and when we discussed about SR-IOV to be in Placement, we said 'well, why not, but it's a mid-term issue' | |
| 09:29:37 | bauzas | at least we first need to do the above before ^ | |
| 09:29:55 | bauzas | so, even if we would like to do it for Victoria, it wouldn't be possible | |
| 09:30:30 | bauzas | Yumeng: that's why I just want to be honest to say that I'm not sure we could do anything for you during Victoria | |
| 09:30:47 | bauzas | and then, not sure we would need to ask all the contributors to be around just for this | |
| 09:31:03 | bauzas | sorry, 'this' being the virtual PTG | |
| 09:31:57 | bauzas | Yumeng: what we *could* do tho is to discuss about that once you provide some docs, maybe just an etherpad for explaining what you'd like | |
| 09:32:36 | Yumeng | bauzas,gibi: the use case is for smartnic and the main integration feature is to pass device_profile from nova to neutron to create a port. | |
| 09:34:58 | openstackgerrit | Huaqiang Wang proposed openstack/nova-specs master: Use PCPU and VCPU in one instance https://review.opendev.org/725801 | |
| 09:35:20 | bauzas | Yumeng: that's not an usecase, it's rather a possible solution ;) | |
| 09:35:37 | bauzas | Yumeng: why do you need to pass this argument to Neutron ? | |
| 09:39:34 | Yumeng | I think this doc makes sense: https://etherpad.opendev.org/p/Shanghai-Neutron-Cyborg-xproj | |
| 09:40:38 | Yumeng | please check from line 41 to end. | |
| 09:42:24 | Yumeng | and line 46 describes why we need to pass the device_profile argument to neutron | |
| 09:42:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix ut error on Aarch64(And other non-x86 platform) https://review.opendev.org/713163 | |
| 09:46:09 | gibi | Yument so a smartnic provides both a programable device and an SRIOV NIC at the same time and the end user would like to specify what program should be loaded to the smarnic and then use that program via a SRIOV device | |
| 09:46:27 | gibi | is this a correct view? | |
| 09:48:52 | Yumeng | gibi: yes,exactly. I think your are right. | |
| 09:52:30 | gibi | bauzas: does this help you approaching the problem? | |
| 09:52:49 | bauzas | gibi: I'm literrally reading the etherpad | |
| 09:53:06 | bauzas | gibi: but I don't get tho why nova needs to tell Neutron what Cyborg wants | |
| 09:53:29 | bauzas | we already call Cyborg at boot time | |
| 09:53:37 | bauzas | so Cyborg could ping Neutron | |
| 09:54:17 | bauzas | the less informations Nova would have to carry, the better I think it would be :) | |
| 09:55:17 | bauzas | also, with us stuck with neutron late binding, this would require subtantial change in nova IIUC | |
| 09:55:21 | gibi | bauzas: current SRIOV support works by creating a port in neutron and providing that port in the nova boot request. if cyborg creates the port in neutron during the boot then we need a way in nova to know about such port | |
| 09:55:41 | bauzas | gibi: I don't want to change this | |
| 09:55:58 | bauzas | gibi: I'm just saying there is a piece of metadata information that neutron has to know | |
| 09:56:04 | bauzas | which is carried by Nova | |
| 09:56:21 | bauzas | but which could be provided by Cyborg before the nova port creation call | |
| 09:56:26 | gibi | sure. I'm not suggesting to change this. I just noted down the logical problem about letting cyborg handle neutron | |
| 09:56:44 | gibi | bauzas: nova does not create the neutron port for sriov | |
| 09:56:58 | gibi | those ports need to be pre-created and provided in the boot request to nova | |
| 09:57:05 | gibi | afaik | |
| 09:57:10 | bauzas | oh you're right | |
| 09:57:15 | bauzas | we don't late bind | |
| 09:57:22 | bauzas | actually, we bind | |
| 09:57:26 | bauzas | but we don't create | |
| 10:05:26 | Yumeng | bauzas: emm IMHO, cyborg doesn't create the port in neutron during the boot. When Cyborg discovers SRIOV device and stores the device's info, user need to use Cyborg create a port/bandwith request, i.e. device_profile | |
| 10:05:56 | bauzas | yeah I was confused, my apologies | |
| 10:06:03 | bauzas | so, there is a pre-creation step | |
| 10:06:14 | bauzas | and then nova gets the port at boot time | |
| 10:06:25 | Yumeng | yes | |
| 10:10:03 | bauzas | OK, I think I still need to go thru the etherpad to understand the reasoning but that's a good first step | |
| 10:10:49 | Yumeng | gibi,bauzas: Thanks for your time! I am going off line now. my time is UTC+8. You can either ping me in IRC at #openstack-cyborg channel if you have more question. or just leave comments at ehterpad(topic 13):https://etherpad.opendev.org/p/cyborg-victoria-goals https://etherpad.opendev.org/p/cyborg-victoria-goals | |
| 10:11:04 | Yumeng | bauzas: thanks for taking time reading it! | |
| 10:11:10 | gibi | Yumeng: thanks I tried to add some not the nova etherpad. | |
| 10:11:17 | bauzas | gibi: fwiw, playing with stestr run --until-failure and then stestr --last helps me reproduce the failure everytime | |
| 10:11:52 | Yumeng | gibi: ok, thanks. I will go back to nova ehterpad later. | |
| 10:12:12 | gibi | bauzas: meantime I was able to reproduce the problem only runnig the single test case test_multiple_instance_create$ so it is not test case interference | |
| 10:12:35 | gibi | (it took 356 runs though) | |
| 10:12:51 | gibi | Yumeng: OK. have a nice evening | |
| 10:12:54 | bauzas | you mean, the single test method() without nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_multiple_instance_create_filling_up_capacity ? | |
| 10:13:02 | bauzas | Yumeng: ack, will do | |
| 10:13:23 | gibi | bauzas: yes a single test method nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_multiple_instance_create [7.793940s] ... FAILED | |
| 10:13:31 | bauzas | gibi: if so, the race is probably because we need to hold for a few things | |
| 10:14:05 | gibi | bauzas: yeah it is most likele a missing wait_for_<something> at some point in the test | |