| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-28 | |||
| 09:01:25 | zigo | reading ... | |
| 09:03:50 | zigo | gibi: I very much agree with all you wrote, thanks you ! | |
| 09:04:00 | zigo | That's exactly what I would like to happen. | |
| 09:04:16 | gibi | zigo: let's see what others think about the feasibility | |
| 09:04:51 | gibi | as I have close to zero knowledge about the generator | |
| 09:18:49 | gibi | bauzas: how the vpgus are avoiding the problem what songwenping_ described for accelerators? | |
| 09:21:16 | gibi | songwenping_: do I understand correctly that your bug appers when you create two instances with one singe POST /servers request? | |
| 09:21:28 | bauzas | gibi: which problem, sorry ? | |
| 09:21:38 | songwenping_ | yes, gibi. | |
| 09:22:41 | bauzas | gibi: do I have a bug to lookup ? | |
| 09:23:19 | gibi | bauzas: same bug you looked at this morning https://bugs.launchpad.net/nova/+bug/1874664 | |
| 09:23:19 | openstack | Launchpad bug 1874664 in OpenStack Compute (nova) "Boot more than one instances failed with accelerators in its flavor" [Medium,Confirmed] - Assigned to Wenping Song (wenping1) | |
| 09:23:21 | gibi | songwenping_: thanks | |
| 09:23:28 | bauzas | hah | |
| 09:24:32 | bauzas | gibi: for vGPUs, we only ask for one RC | |
| 09:24:51 | bauzas | eg. resources:VGPU=2 | |
| 09:25:31 | bauzas | so, when calling Placement, it returns some allocations but all the VGPU ones are in the same RP | |
| 09:25:54 | gibi | bauzas: in case of instance multi create you need two separate allocation of one VGPUs | |
| 09:26:10 | gibi | bauzas: I mean instance multi create with min=2 max=2 | |
| 09:26:15 | bauzas | ah this | |
| 09:27:01 | bauzas | gibi: actually https://bugs.launchpad.net/nova/+bug/1780225 :-) | |
| 09:27:01 | openstack | Launchpad bug 1780225 in OpenStack Compute (nova) "Libvirt error when using --max > 1 with vGPU" [High,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 09:27:55 | gibi | bauzas: ohh but that leads to a different error in libvirt instead of placement | |
| 09:28:22 | bauzas | gibi: I should be testing it | |
| 09:28:37 | bauzas | lemme try to use the functional test | |
| 09:29:57 | lyarwood | does anyone have an example in api-ref where we list some pre-conditions to calling an API? Looking at the evacuate section I wanted to add a pre-condition around the host being fenced and service being reported as down/forced-down. | |
| 09:30:58 | lyarwood | ah found some for reboot nvm | |
| 09:32:04 | gibi | bauzas: can we have two PGPU RPs providing the same vgpu types under the same compute RP? If yes then I think the instance multicreate can fail due to running out of VGPU resource on the PGPU in the first allocation candidate | |
| 09:32:36 | gibi | and that would be similar to what songwenping_ reported for accelerators | |
| 09:32:46 | bauzas | gibi: yes of course | |
| 09:33:42 | bauzas | gibi: what do you want me to test ? | |
| 09:34:06 | bauzas | 2 RPs with the same vgpu type, each them having, say, 8 vGPUs | |
| 09:34:22 | bauzas | then asking for 2 instances for 2 vGPUs | |
| 09:34:28 | bauzas | gibi: works with you ? | |
| 09:34:41 | gibi | bauzas: give me a sec | |
| 09:35:16 | bauzas | gibi: fwiw I'm just trying to reproduce https://bugs.launchpad.net/nova/+bug/1780225 with the functional tests | |
| 09:35:16 | openstack | Launchpad bug 1780225 in OpenStack Compute (nova) "Libvirt error when using --max > 1 with vGPU" [High,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 09:37:32 | gibi | bauzas: you could try that case when ask for 2 instance with 2 vGPUs each, then the compute has two PGPUs 2 VGPU each | |
| 09:38:03 | aarents | lyarwood: just FYI http://paste.openstack.org/show/792735/ | |
| 09:39:06 | gibi | I now assume that the multi create will try to allocate the resources in placement for the second intance from the same PGPU as for the first instance due to https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L238 | |
| 09:39:43 | gibi | bauzas: as we are never considering the second allocation candidate from the a host | |
| 09:39:53 | gibi | s/considering/consider/ | |
| 09:39:54 | bauzas | gibi: okay, lemme try to do it | |
| 09:39:58 | gibi | OK | |
| 09:40:20 | gibi | bauzas: in general I agree that the songwenping_'s bug is not an RC2 candidate. Let's just document it | |
| 09:40:22 | bauzas | gibi: /me just looks how to simulate multi create with functests | |
| 09:41:37 | gibi | bauzas: test_create_multiple_servers | |
| 09:41:41 | gibi | bauzas: https://github.com/openstack/nova/blob/be9afebd6a6c400d2c7f7a13dfe25c8c641c1baa/nova/tests/functional/test_servers.py#L589 | |
| 09:41:48 | bauzas | thanks, I know we had it | |
| 09:41:56 | bauzas | but I was trying to find the test :) | |
| 09:42:10 | bauzas | test_servers <3 | |
| 09:42:31 | openstackgerrit | Lee Yarwood proposed openstack/nova master: docs: Add evacuation pre-conditions around the src host https://review.opendev.org/723853 | |
| 09:42:50 | lyarwood | aarents: ack thanks | |
| 09:43:38 | lyarwood | aarents: so it's always adding that additional block | |
| 09:45:26 | lyarwood | aarents: ah sorry you snipped the output | |
| 09:45:31 | lyarwood | aarents: so sometimes it did? | |
| 09:45:36 | aarents | lyarwood: not always.. that's odd, in the paste only 7% of times and it depend of size you fallocate | |
| 09:45:41 | aarents | lyarwood: yep | |
| 09:45:45 | lyarwood | aarents: right my bad | |
| 09:45:48 | lyarwood | aarents: yeah that's weird | |
| 09:47:30 | gibi | brinzhang, songwenping_ : so I assume that some of you will propose patach that documents the limitation about multi create with accelerators | |
| 09:47:39 | bauzas | grrrr, /me raises hand at tox.ini when you change dependencies and then you only have DSL 7Mbps bandwidth | |
| 09:47:55 | gibi | brinzhang_, songwenping_: let me know if you need help | |
| 09:56:01 | bauzas | gibi: interesting, when trying to multi create 2 instances asking for 1 vGPU each, I get no exceptions but it tells me 4 vGPUs instead of 2 | |
| 09:57:10 | bauzas | (one compute, 2 pGPUs with a capacity of 8 each) | |
| 09:57:22 | brinzhang | gibi: later, we will propose a patch, to note the multi create in https://docs.openstack.org/api-guide/compute/accelerator-support.html | |
| 09:58:56 | bauzas | gibi: ahah, no, that's just my test which is wrong, we create 2 mdevs :p | |
| 10:00:36 | bauzas | gibi: so, I can't reproduce https://bugs.launchpad.net/nova/+bug/1780225 with my functional test | |
| 10:00:36 | openstack | Launchpad bug 1780225 in OpenStack Compute (nova) "Libvirt error when using --max > 1 with vGPU" [High,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 10:05:03 | bauzas | given it was for Rocky, I think it was fixed by the fact that in Stein we have a RP per pGPU | |
| 10:13:27 | gibi | brinzhang: ack, thanks a lot | |
| 10:14:26 | brinzhang | gibi: np | |
| 10:15:10 | gibi | bauzas: do you try that with the libvirt driver or with the fake driver? I think the placement issue can be recreated with the fake driver | |
| 10:15:41 | bauzas | gibi: i use the fakelibvirt driver | |
| 10:15:52 | bauzas | not the fake driver itself | |
| 10:15:57 | gibi | I see | |
| 10:16:36 | gibi | anyhow it is not burning hot issue right now and songwenping_ PoC seems to be a step in the good direction | |
| 10:17:52 | gibi | as soon as we have a func recreate for the accelerator case we can adapt that for the VGPU case as well (with the fake driver) | |
| 10:24:02 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Test multi create with vGPUs https://review.opendev.org/723858 | |
| 10:24:08 | bauzas | gibi: ^ | |
| 10:24:31 | gibi | bauzas: ack, I will look | |
| 10:24:57 | bauzas | gibi: I'll provide a new revision for it with testing what happens when you ask for 2 vGPUs in multicreate but each pGPU can only create one | |
| 10:25:14 | kashyap | lyarwood: Hi, I'm just digging into the Q35 failure with 'virt-preview': seems like the logs are already gone | |
| 10:25:19 | gibi | bauzas: OK | |
| 10:26:08 | bauzas | gibi: tbc, placement doesn't support sharding resources over RPs but here this is not the issue | |
| 10:26:13 | kashyap | lyarwood: I take it that you haven't had a chance to look at them I just did a 'recheck' for it to reun | |
| 10:26:17 | kashyap | s/reun/rerun/ | |
| 10:27:45 | gibi | bauzas: yeah, if we have two instance requesting one VGPU each then we never allocate two VGPUs in the same placement request so no sharding is needed from placement | |
| 10:34:51 | lyarwood | kashyap: I haven't sorry | |
| 10:35:18 | bauzas | gibi: interesting, we don't have the problem for multi-create with vGPUs filling up capacity | |
| 10:39:19 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Test multi create with vGPUs https://review.opendev.org/723858 | |
| 10:39:36 | bauzas | gibi: see above, I created two instances with 8 vGPUs each | |
| 10:39:54 | kashyap | yarwood: No problem; I'll dig in | |
| 10:42:27 | bauzas | oh snap | |
| 10:43:21 | bauzas | (I just ran the older test and now the new one) | |
| 10:45:40 | bauzas | ah, reproduced | |
| 11:04:32 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Test multi create with vGPUs https://review.opendev.org/723858 | |
| 11:05:01 | bauzas | brinzhang: songwenping_: gibi: confirmed the issue for vGPUs | |
| 11:05:16 | bauzas | see the above patch https://review.opendev.org/723858 | |
| 11:05:34 | bauzas | so the problem is not related to cyborg but rather for all nested resource providers | |