Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-07
17:26:43 gibi another question, does the series gracefully handle that move operations are not supported yet with accelerators?
17:26:55 stephenfin efried: I think earliest-version would do the trick. Let me try with that
17:29:09 dansmith gibi: probably not, but I'm trying to think of other examples of things we know don't work where we handle that gracefully
17:29:32 gibi dansmith: qos ports was implemented a check at the API level
17:29:34 dansmith gibi: there are some cases where we don't really know until the virt driver(s) get involved, which is *kinda* this case
17:30:07 gibi dansmith: here we know that nova does not support migrating an instance with arq yet, so we should reject that
17:30:11 dansmith gibi: okay but that's kindof a high-level thing because the coordination needs to be done above the compute (i.e. with neutron) anyway right?
17:30:29 dansmith gibi: you mean libvirt I assume
17:31:01 stephenfin efried: nope, still a couple of minutes and fans running at full blast :)
17:31:34 dansmith well, hmm, I was going to say the nova bits would still try, but maybe not because the conductor kicks the bind.. /me looks
17:31:40 gibi I think not just libvirt but also nova needs to grow support for migration with arqs, like re-querying the resource request of the arq at the start of the migration to include them into the scheduling request
17:32:52 dansmith gibi: yeah I guess you're right since we don't persist those in the reqspec
17:33:32 gibi OK, left this as well as a comment in https://review.opendev.org/#/c/631244/61
17:35:06 gibi I have to leave soon I will continue reading the series on Monday
17:35:34 dansmith I guess we probably have cold migration handling for pci devices specifically,
17:35:57 dansmith but I was kinda thinking this would fail for live migration like I think it does for pci, which is.. late in the virt driver IIRC
17:37:37 gibi my point is that we know we need to write some nova code to support these ops. So while we dont have that code we can reject such ops from the API.
17:38:21 dansmith gibi: yep, agree, I'm just talking out loud
17:38:30 dansmith talking out loud? thinking out loud :)
17:39:18 dansmith I don't want to put api-level barriers to something that is just a virt limitation but you're right, none of the paths where we do the arq stuff gets tickled in the, for example, resize paths
17:40:11 gibi yeah, I thin resize, migrate, live migrate, evacuate, unshelve (after offload) needs some code to recreate the proper resource request for the scheduling
17:40:41 dansmith yup
17:42:09 dansmith gibi: I was telling efried the other day that I had been so tunnel-vision on the bones of this that I wanted him to run through it again to break that up for me
17:42:18 dansmith for exactly this reason, so thanks for being that force :)
17:44:39 gibi I saw that two of you already handling this seris so I decided to focus energies elsewhere. but agree that fresh eyes helps to se things differently
17:48:16 dansmith yup
18:10:41 spatel sean-k-mooney: I am back now, sorry was in back to back meeting
18:38:05 sean-k-mooney dansmith: we have a check that blocks live migration if there are pci_request spec object that are not related to neutorn sriov ports
18:38:18 sean-k-mooney dansmith: but that wont block cyborg devices
18:38:25 dansmith sean-k-mooney: okay so something similar for cyborg I guess
18:38:31 sean-k-mooney since we also dont have pci_request spec objects
18:39:04 sean-k-mooney well for now we could just check if the flavor has accl:device-profile or whatever teh extraspec is
18:39:18 dansmith right
18:40:22 sean-k-mooney im not sure we will be able to do that check in the api because we would need to check the embeded flaovr form the cell db. actully no we can. we can grab it from the request_spec in the api db
18:40:42 sean-k-mooney so ya we can reject the live migration in the api layer until we support that
18:41:07 dansmith not sure what the problem is.. the api can look at the instance's embedded flavor
18:41:30 sean-k-mooney yep it can so no problem
18:41:45 sean-k-mooney i was thinking we might need to do a down call to the cell db to get it but we dont
18:41:49 sean-k-mooney so all good
18:42:06 dansmith we do to get the actual embedded flavor, but that's fine of copurse
18:42:21 dansmith anything that does anything on an instance gets the instance record from the cell in the api
18:42:28 sean-k-mooney well the embeded flavor is aslo stored in the api db in the request spec
18:42:53 dansmith it's stored as it was at the time of creation, but not necessarily the same as what the instance has now
18:43:04 dansmith and if we've done a data migration or something they could have diverged
18:43:11 sean-k-mooney oh that wont be update after reisze?
18:43:18 sean-k-mooney ah ok
18:43:25 dansmith the instance's actual flavor is what we should use, and it's no more expensive to get
18:43:44 sean-k-mooney sure makes sense
18:45:16 mriedem the requestspec.flavor is updated as part of a resize, but .... there be bugs
18:50:04 spatel sean-k-mooney: here is the virsh capability - http://paste.openstack.org/show/789301/
18:51:00 spatel here is the vCPU pinning map - http://paste.openstack.org/show/789302/
18:58:23 sean-k-mooney ok so the first 15 cpus are pinned to host numa node 0
18:58:33 spatel Yes
18:58:59 sean-k-mooney and the second 15 are all pinned to host numa node 1
18:59:02 sean-k-mooney so this is correct
18:59:06 spatel Yes
18:59:51 spatel if you see all looks correct from VM point of view..
19:00:21 spatel VM correctly mapped its vCPU pins across physical numa nodes
19:00:43 sean-k-mooney yes so you left out one of the imporatn numa elelmnt form the vm
19:00:59 spatel ?
19:01:10 sean-k-mooney you left out the numa element
19:01:11 mriedem important numa elements from the vm
19:01:20 mriedem translation-as-a-service
19:01:35 sean-k-mooney oh ya i miss typed talking downstream as well
19:01:36 spatel I am not following you guys..
19:01:39 sean-k-mooney ill slow down
19:01:56 mriedem remember how this isn't a support channel?
19:02:30 sean-k-mooney nova will map the first half of the cpus to numa node 0 in the guess so cores 0-15 and form what you have show it is working correctly
19:03:05 spatel Yes.. that is what i am also saying.. CPU pins correctly mapped out to NUMA nodes
19:03:52 spatel This is what NUMA looks inside VM0 -> http://paste.openstack.org/show/789304/
19:04:21 spatel now question is when i run application why i am seeing poor performance?
19:04:50 Sundar gibi: Any pointers to how you handled reschedule for bandwidth provider?
19:04:56 spatel Do you think i need to do something with -> hw:cpu_thread_policy ?
19:05:28 sean-k-mooney spatel: it woudl appear the aplication is not correctly optimising for the numa toplogy.
19:05:41 sean-k-mooney hw:cpu_thread_polciy wont help
19:06:23 sean-k-mooney you can alther the cpu thread and socket toplogy in the guest which might but this does not seeam to be a nova/libvirt issue
19:07:12 sean-k-mooney spatel: my guess is that you have not toled the guest it is using hyper treads so it is miss allocationg threads internally to cores it think are independed but are not
19:07:33 spatel I am just trying to understand if i run application on bare metal then performance is great but when i create VM with all CPU then performance is bad..
19:07:36 sean-k-mooney hw:cpu_thread_policy=isolate woudl help but it woudl reduce the available core count
19:08:06 sean-k-mooney you shoudl set hw:cpu_treads=2 and hw:cpu_sockets=2
19:08:11 spatel how may core count it will reduce ?
19:08:16 sean-k-mooney but you cant do that with 30 cpus
19:08:26 sean-k-mooney you need either 28 or 32
19:08:35 spatel i will go with 28..
19:08:39 Sundar gibi: "does the series gracefully handle that move operations are not supported yet with accelerators?" There was an earlier attempt to explicitly block unsupported ops: https://review.opendev.org/#/c/674726/ But it was decided that it is not needed, and we should document the supported ops in Cyborg.
19:08:39 sean-k-mooney so that you have an even number of cpus per numa node/socket
19:09:20 spatel you are saying i should set flavor -> hw:cpu_treads=2 and hw:cpu_sockets=2 and run benchmark right?
19:09:35 sean-k-mooney spatel: set hw:cpu_treads=2 and hw:cpu_sockets=2 and do not set hw:cpu_thread_policy=isolate and see if that fixes the issue
19:09:48 spatel sean-k-mooney: sounds good
19:10:12 sean-k-mooney spatel: actuly i misspelled the first one check the flavor docs to confrim they are right
19:10:27 sean-k-mooney spatel: https://docs.openstack.org/nova/latest/user/flavors.html
19:10:37 spatel I will check doc don't worry all i need clue to try something :)
19:11:17 Sundar gibi: "nova needs to grow support for migration with arqs" Since Cyborg does PCI passthrough, live migration is not supported.
19:11:40 spatel sean-k-mooney: you are saying that option will tell my VM you have two socket..
19:11:41 sean-k-mooney by default openstack emulates each gust cpu as a sperate socket which is incorrect if you have HT enabled on the host
19:11:54 spatel I do have HT enabled
19:12:17 sean-k-mooney yep so you want to tell the vm it has 2 threads per cpu and in this case 2 socket 1 per numa node
19:12:34 sean-k-mooney that will help the guest kerenl make correct schduling decisions
19:13:03 spatel sean-k-mooney: i think you are correct.. that could be the issue why erlang getting confused
19:13:19 spatel because erlang also run inside VM and it should have own scheduler

Earlier   Later