Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-07
09:46:09 gibi Yument so a smartnic provides both a programable device and an SRIOV NIC at the same time and the end user would like to specify what program should be loaded to the smarnic and then use that program via a SRIOV device
09:46:27 gibi is this a correct view?
09:48:52 Yumeng gibi: yes,exactly. I think your are right.
09:52:30 gibi bauzas: does this help you approaching the problem?
09:52:49 bauzas gibi: I'm literrally reading the etherpad
09:53:06 bauzas gibi: but I don't get tho why nova needs to tell Neutron what Cyborg wants
09:53:29 bauzas we already call Cyborg at boot time
09:53:37 bauzas so Cyborg could ping Neutron
09:54:17 bauzas the less informations Nova would have to carry, the better I think it would be :)
09:55:17 bauzas also, with us stuck with neutron late binding, this would require subtantial change in nova IIUC
09:55:21 gibi bauzas: current SRIOV support works by creating a port in neutron and providing that port in the nova boot request. if cyborg creates the port in neutron during the boot then we need a way in nova to know about such port
09:55:41 bauzas gibi: I don't want to change this
09:55:58 bauzas gibi: I'm just saying there is a piece of metadata information that neutron has to know
09:56:04 bauzas which is carried by Nova
09:56:21 bauzas but which could be provided by Cyborg before the nova port creation call
09:56:26 gibi sure. I'm not suggesting to change this. I just noted down the logical problem about letting cyborg handle neutron
09:56:44 gibi bauzas: nova does not create the neutron port for sriov
09:56:58 gibi those ports need to be pre-created and provided in the boot request to nova
09:57:05 gibi afaik
09:57:10 bauzas oh you're right
09:57:15 bauzas we don't late bind
09:57:22 bauzas actually, we bind
09:57:26 bauzas but we don't create
10:05:26 Yumeng bauzas: emm IMHO, cyborg doesn't create the port in neutron during the boot. When Cyborg discovers SRIOV device and stores the device's info, user need to use Cyborg create a port/bandwith request, i.e. device_profile
10:05:56 bauzas yeah I was confused, my apologies
10:06:03 bauzas so, there is a pre-creation step
10:06:14 bauzas and then nova gets the port at boot time
10:06:25 Yumeng yes
10:10:03 bauzas OK, I think I still need to go thru the etherpad to understand the reasoning but that's a good first step
10:10:49 Yumeng gibi,bauzas: Thanks for your time! I am going off line now. my time is UTC+8. You can either ping me in IRC at #openstack-cyborg channel if you have more question. or just leave comments at ehterpad(topic 13):https://etherpad.opendev.org/p/cyborg-victoria-goals https://etherpad.opendev.org/p/cyborg-victoria-goals
10:11:04 Yumeng bauzas: thanks for taking time reading it!
10:11:10 gibi Yumeng: thanks I tried to add some not the nova etherpad.
10:11:17 bauzas gibi: fwiw, playing with stestr run --until-failure and then stestr --last helps me reproduce the failure everytime
10:11:52 Yumeng gibi: ok, thanks. I will go back to nova ehterpad later.
10:12:12 gibi bauzas: meantime I was able to reproduce the problem only runnig the single test case test_multiple_instance_create$ so it is not test case interference
10:12:35 gibi (it took 356 runs though)
10:12:51 gibi Yumeng: OK. have a nice evening
10:12:54 bauzas you mean, the single test method() without nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_multiple_instance_create_filling_up_capacity ?
10:13:02 bauzas Yumeng: ack, will do
10:13:23 gibi bauzas: yes a single test method nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_multiple_instance_create [7.793940s] ... FAILED
10:13:31 bauzas gibi: if so, the race is probably because we need to hold for a few things
10:14:05 gibi bauzas: yeah it is most likele a missing wait_for_<something> at some point in the test
10:14:07 bauzas gibi: lemme see if adding a pause of 1 sec will just solve the problem
10:14:18 gibi ack
10:14:24 bauzas gibi: because what's failing is that we lookup the mdevs
10:14:34 bauzas gibi: we should potentially wait a bit
10:15:02 bauzas this being said, we wait for the ACTIVE status
10:15:22 bauzas oh wait
10:15:30 bauzas this is a multiple instance creation
10:15:43 bauzas so we only check the first instance creation
10:16:05 gibi that could explain it
10:16:19 bauzas lemme try
10:16:38 bauzas --until-failure allows me to run the test easily
10:18:07 bauzas ahah, interesting
10:18:11 bauzas we don't hold
10:26:21 bauzas gibi: okay, changed my test to wait for all servers, running it
10:26:28 bauzas gibi: will leave it run for one hour or so
10:26:28 gibi cool
10:26:44 gibi if you push the patch up I can do the same while I have lunch
10:26:54 bauzas and if I no longer see any issues after, say, 200 runs, then we will know it's fixed
10:27:01 bauzas k
10:27:28 bauzas gibi: just writing a bug and uploading then
10:28:09 gibi ack
10:29:36 bauzas ah shit, they started it :p
10:30:44 openstack Launchpad bug 1877281 in OpenStack Compute (nova) "vGPU multiple instance creation test is racey" [High,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza)
10:30:44 bauzas https://bugs.launchpad.net/nova/+bug/1877281
10:31:27 bauzas gibi: worth thinking about RC3 for this one ? :(
10:31:50 bauzas or, maybe we just release and then backport later
10:31:59 bauzas the issue isn't that bad
10:32:12 jazeltq hello gibi
10:32:41 jazeltq I have one question about nova-scheduler
10:35:56 openstackgerrit Sylvain Bauza proposed openstack/nova master: Wait for all servers to be active when testing vGPUs https://review.opendev.org/726087
10:36:19 bauzas gibi: ^
10:37:01 bauzas gibi: FYI (functional) [sbauza@sbauza nova]$ stestr --test-path=./nova/tests/functional run --until-failure nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_multiple_instance_create
10:42:21 stephenfin melwitt: When you're about, can you take a look at https://review.opendev.org/#/c/677444/ please?
10:48:39 gibi bauzas: I don't think we need an RC for a racy func test. It is OK to merge it after GA
12:00:36 bauzas gibi: okay, after 4560 seconds, no errors were found
12:00:57 gibi bauzas: no error seen in 412 runs
12:01:02 gibi so I agree you fixed it!
12:01:28 gibi nice work
12:27:54 efried stephenfin: go ahead and update that commit message. I'm here to re+1, for all the good it'll do ya.
12:35:25 openstackgerrit Stephen Finucane proposed openstack/nova master: config: Explicitly register 'remote_debug' CLI opts https://review.opendev.org/677444
12:35:31 openstack Launchpad bug 1877281 in OpenStack Compute (nova) "vGPU multiple instance creation test is racey" [High,In progress] - Assigned to Sylvain Bauza (sylvain-bauza)
12:35:31 bauzas stephenfin: https://review.opendev.org/#/c/726087/1 closing https://bugs.launchpad.net/nova/+bug/1877281
12:35:32 stephenfin efried: ack, thanks
12:35:39 stephenfin bauzas++
12:37:55 efried stephenfin: done (with bonus)
12:39:48 stephenfin bauzas: I have an ask on https://review.opendev.org/#/c/726087/
12:40:01 stephenfin if you don't want to, I can live with it as-is
12:43:59 bauzas stephenfin: no worries, I can write a new rev
12:44:06 bauzas there is no urgency to fix this race
12:44:18 bauzas we only get it for around 0.5%
12:45:55 openstackgerrit Sylvain Bauza proposed openstack/nova master: Wait for all servers to be active when testing vGPUs https://review.opendev.org/726087
12:46:10 bauzas stephenfin: fer your plizure ^
12:46:47 stephenfin +"
12:46:49 stephenfin *2
13:18:57 francoisp bauzas, salut, when you have time: https://review.opendev.org/#/c/725272/ (this time for Rocky) - thanks
14:27:17 bauzas francoisp: ack, will try to do it asap
14:27:43 francoisp bauzas, thank you!
14:32:53 openstackgerrit Jiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD https://review.opendev.org/572805

Earlier   Later