Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-04-28
09:29:57 lyarwood does anyone have an example in api-ref where we list some pre-conditions to calling an API? Looking at the evacuate section I wanted to add a pre-condition around the host being fenced and service being reported as down/forced-down.
09:30:58 lyarwood ah found some for reboot nvm
09:32:04 gibi bauzas: can we have two PGPU RPs providing the same vgpu types under the same compute RP? If yes then I think the instance multicreate can fail due to running out of VGPU resource on the PGPU in the first allocation candidate
09:32:36 gibi and that would be similar to what songwenping_ reported for accelerators
09:32:46 bauzas gibi: yes of course
09:33:42 bauzas gibi: what do you want me to test ?
09:34:06 bauzas 2 RPs with the same vgpu type, each them having, say, 8 vGPUs
09:34:22 bauzas then asking for 2 instances for 2 vGPUs
09:34:28 bauzas gibi: works with you ?
09:34:41 gibi bauzas: give me a sec
09:35:16 openstack Launchpad bug 1780225 in OpenStack Compute (nova) "Libvirt error when using --max > 1 with vGPU" [High,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza)
09:35:16 bauzas gibi: fwiw I'm just trying to reproduce https://bugs.launchpad.net/nova/+bug/1780225 with the functional tests
09:37:32 gibi bauzas: you could try that case when ask for 2 instance with 2 vGPUs each, then the compute has two PGPUs 2 VGPU each
09:38:03 aarents lyarwood: just FYI http://paste.openstack.org/show/792735/
09:39:06 gibi I now assume that the multi create will try to allocate the resources in placement for the second intance from the same PGPU as for the first instance due to https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L238
09:39:43 gibi bauzas: as we are never considering the second allocation candidate from the a host
09:39:53 gibi s/considering/consider/
09:39:54 bauzas gibi: okay, lemme try to do it
09:39:58 gibi OK
09:40:20 gibi bauzas: in general I agree that the songwenping_'s bug is not an RC2 candidate. Let's just document it
09:40:22 bauzas gibi: /me just looks how to simulate multi create with functests
09:41:37 gibi bauzas: test_create_multiple_servers
09:41:41 gibi bauzas: https://github.com/openstack/nova/blob/be9afebd6a6c400d2c7f7a13dfe25c8c641c1baa/nova/tests/functional/test_servers.py#L589
09:41:48 bauzas thanks, I know we had it
09:41:56 bauzas but I was trying to find the test :)
09:42:10 bauzas test_servers <3
09:42:31 openstackgerrit Lee Yarwood proposed openstack/nova master: docs: Add evacuation pre-conditions around the src host https://review.opendev.org/723853
09:42:50 lyarwood aarents: ack thanks
09:43:38 lyarwood aarents: so it's always adding that additional block
09:45:26 lyarwood aarents: ah sorry you snipped the output
09:45:31 lyarwood aarents: so sometimes it did?
09:45:36 aarents lyarwood: not always.. that's odd, in the paste only 7% of times and it depend of size you fallocate
09:45:41 aarents lyarwood: yep
09:45:45 lyarwood aarents: right my bad
09:45:48 lyarwood aarents: yeah that's weird
09:47:30 gibi brinzhang, songwenping_ : so I assume that some of you will propose patach that documents the limitation about multi create with accelerators
09:47:39 bauzas grrrr, /me raises hand at tox.ini when you change dependencies and then you only have DSL 7Mbps bandwidth
09:47:55 gibi brinzhang_, songwenping_: let me know if you need help
09:56:01 bauzas gibi: interesting, when trying to multi create 2 instances asking for 1 vGPU each, I get no exceptions but it tells me 4 vGPUs instead of 2
09:57:10 bauzas (one compute, 2 pGPUs with a capacity of 8 each)
09:57:22 brinzhang gibi: later, we will propose a patch, to note the multi create in https://docs.openstack.org/api-guide/compute/accelerator-support.html
09:58:56 bauzas gibi: ahah, no, that's just my test which is wrong, we create 2 mdevs :p
10:00:36 openstack Launchpad bug 1780225 in OpenStack Compute (nova) "Libvirt error when using --max > 1 with vGPU" [High,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza)
10:00:36 bauzas gibi: so, I can't reproduce https://bugs.launchpad.net/nova/+bug/1780225 with my functional test
10:05:03 bauzas given it was for Rocky, I think it was fixed by the fact that in Stein we have a RP per pGPU
10:13:27 gibi brinzhang: ack, thanks a lot
10:14:26 brinzhang gibi: np
10:15:10 gibi bauzas: do you try that with the libvirt driver or with the fake driver? I think the placement issue can be recreated with the fake driver
10:15:41 bauzas gibi: i use the fakelibvirt driver
10:15:52 bauzas not the fake driver itself
10:15:57 gibi I see
10:16:36 gibi anyhow it is not burning hot issue right now and songwenping_ PoC seems to be a step in the good direction
10:17:52 gibi as soon as we have a func recreate for the accelerator case we can adapt that for the VGPU case as well (with the fake driver)
10:24:02 openstackgerrit Sylvain Bauza proposed openstack/nova master: WIP: Test multi create with vGPUs https://review.opendev.org/723858
10:24:08 bauzas gibi: ^
10:24:31 gibi bauzas: ack, I will look
10:24:57 bauzas gibi: I'll provide a new revision for it with testing what happens when you ask for 2 vGPUs in multicreate but each pGPU can only create one
10:25:14 kashyap lyarwood: Hi, I'm just digging into the Q35 failure with 'virt-preview': seems like the logs are already gone
10:25:19 gibi bauzas: OK
10:26:08 bauzas gibi: tbc, placement doesn't support sharding resources over RPs but here this is not the issue
10:26:13 kashyap lyarwood: I take it that you haven't had a chance to look at them I just did a 'recheck' for it to reun
10:26:17 kashyap s/reun/rerun/
10:27:45 gibi bauzas: yeah, if we have two instance requesting one VGPU each then we never allocate two VGPUs in the same placement request so no sharding is needed from placement
10:34:51 lyarwood kashyap: I haven't sorry
10:35:18 bauzas gibi: interesting, we don't have the problem for multi-create with vGPUs filling up capacity
10:39:19 openstackgerrit Sylvain Bauza proposed openstack/nova master: Test multi create with vGPUs https://review.opendev.org/723858
10:39:36 bauzas gibi: see above, I created two instances with 8 vGPUs each
10:39:54 kashyap yarwood: No problem; I'll dig in
10:42:27 bauzas oh snap
10:43:21 bauzas (I just ran the older test and now the new one)
10:45:40 bauzas ah, reproduced
11:04:32 openstackgerrit Sylvain Bauza proposed openstack/nova master: Test multi create with vGPUs https://review.opendev.org/723858
11:05:01 bauzas brinzhang: songwenping_: gibi: confirmed the issue for vGPUs
11:05:16 bauzas see the above patch https://review.opendev.org/723858
11:05:34 bauzas so the problem is not related to cyborg but rather for all nested resource providers
11:06:37 bauzas anyway, not a RC regression given we had the same issue in Train for vGPUs (and possibly bandwidth-aware instances,wdyt gibi ?)
11:16:31 gibi bauzas: thanks for the reporoduction
11:25:03 openstack Launchpad bug 1875624 in OpenStack Compute (nova) "the vms can not be force deleted when vm_status is soft-delete and task-state=deleting" [Undecided,New] - Assigned to xuyuanhao (thourch)
11:25:03 brinzhang_ gibi: my partner proposed a bug, see https://bugs.launchpad.net/nova/+bug/1875624
11:25:08 brinzhang_ gibi: can you check
11:25:20 gibi brinzhang_: will check soon
11:25:27 brinzhang_ gibi: thanks
11:40:38 gibi brinzhang_: confirmed the bug.
12:30:37 openstackgerrit Brin Zhang proposed openstack/nova master: Add nested resource providers limit for multi create https://review.opendev.org/723884
12:30:37 openstack bug 1874664 in OpenStack Compute (nova) "Boot more than one instances failed with accelerators in its flavor" [Medium,Confirmed] https://launchpad.net/bugs/1874664 - Assigned to Wenping Song (wenping1)
12:30:37 bauzas gibi: brinzhang: FWIW, I think we should change the title of bug 1874664 to make it clear it's for all nested Resource Providers
12:31:01 gibi bauzas: I agree
12:32:59 openstack Launchpad bug 1874664 in OpenStack Compute (nova) "Instance multi-create doesn't support available resources spread between children RPs" [Medium,Confirmed] - Assigned to Wenping Song (wenping1)
12:32:59 bauzas changed https://bugs.launchpad.net/nova/+bug/1874664
12:33:35 bauzas gibi: have you tested it for bandwidth-aware instances ?
12:33:52 bauzas actually, that's traits, right?
12:34:35 gibi bauzas: multi create with neutron ports are not supported even without bandwidth
12:34:50 gibi multi create with neutron net is supported without bandwidth
12:35:00 bauzas ack ok
12:35:33 bauzas fwiw, multi-create works with vGPUs if one RP has all the capacity for all the instances
12:35:49 brinzhang_ bauzas, gibi: I am now confused (dazzled) ^^
12:36:02 bauzas and I guess it's the same for cyborg resources
12:36:47 gibi brinzhang_, bauzas: I have to jump on a call, sorry. I will read back later
12:36:52 brinzhang_ bauzas: you mean change "Add" to "All"?
12:38:09 brinzhang_ bauzas: if possiable, you can edit that patch, or you can leave comments inline, I have to go home now, it's too later for me, I am sorry

Earlier   Later