| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-07 | |||
| 17:30:29 | dansmith | gibi: you mean libvirt I assume | |
| 17:31:01 | stephenfin | efried: nope, still a couple of minutes and fans running at full blast :) | |
| 17:31:34 | dansmith | well, hmm, I was going to say the nova bits would still try, but maybe not because the conductor kicks the bind.. /me looks | |
| 17:31:40 | gibi | I think not just libvirt but also nova needs to grow support for migration with arqs, like re-querying the resource request of the arq at the start of the migration to include them into the scheduling request | |
| 17:32:52 | dansmith | gibi: yeah I guess you're right since we don't persist those in the reqspec | |
| 17:33:32 | gibi | OK, left this as well as a comment in https://review.opendev.org/#/c/631244/61 | |
| 17:35:06 | gibi | I have to leave soon I will continue reading the series on Monday | |
| 17:35:34 | dansmith | I guess we probably have cold migration handling for pci devices specifically, | |
| 17:35:57 | dansmith | but I was kinda thinking this would fail for live migration like I think it does for pci, which is.. late in the virt driver IIRC | |
| 17:37:37 | gibi | my point is that we know we need to write some nova code to support these ops. So while we dont have that code we can reject such ops from the API. | |
| 17:38:21 | dansmith | gibi: yep, agree, I'm just talking out loud | |
| 17:38:30 | dansmith | talking out loud? thinking out loud :) | |
| 17:39:18 | dansmith | I don't want to put api-level barriers to something that is just a virt limitation but you're right, none of the paths where we do the arq stuff gets tickled in the, for example, resize paths | |
| 17:40:11 | gibi | yeah, I thin resize, migrate, live migrate, evacuate, unshelve (after offload) needs some code to recreate the proper resource request for the scheduling | |
| 17:40:41 | dansmith | yup | |
| 17:42:09 | dansmith | gibi: I was telling efried the other day that I had been so tunnel-vision on the bones of this that I wanted him to run through it again to break that up for me | |
| 17:42:18 | dansmith | for exactly this reason, so thanks for being that force :) | |
| 17:44:39 | gibi | I saw that two of you already handling this seris so I decided to focus energies elsewhere. but agree that fresh eyes helps to se things differently | |
| 17:48:16 | dansmith | yup | |
| 18:10:41 | spatel | sean-k-mooney: I am back now, sorry was in back to back meeting | |
| 18:38:05 | sean-k-mooney | dansmith: we have a check that blocks live migration if there are pci_request spec object that are not related to neutorn sriov ports | |
| 18:38:18 | sean-k-mooney | dansmith: but that wont block cyborg devices | |
| 18:38:25 | dansmith | sean-k-mooney: okay so something similar for cyborg I guess | |
| 18:38:31 | sean-k-mooney | since we also dont have pci_request spec objects | |
| 18:39:04 | sean-k-mooney | well for now we could just check if the flavor has accl:device-profile or whatever teh extraspec is | |
| 18:39:18 | dansmith | right | |
| 18:40:22 | sean-k-mooney | im not sure we will be able to do that check in the api because we would need to check the embeded flaovr form the cell db. actully no we can. we can grab it from the request_spec in the api db | |
| 18:40:42 | sean-k-mooney | so ya we can reject the live migration in the api layer until we support that | |
| 18:41:07 | dansmith | not sure what the problem is.. the api can look at the instance's embedded flavor | |
| 18:41:30 | sean-k-mooney | yep it can so no problem | |
| 18:41:45 | sean-k-mooney | i was thinking we might need to do a down call to the cell db to get it but we dont | |
| 18:41:49 | sean-k-mooney | so all good | |
| 18:42:06 | dansmith | we do to get the actual embedded flavor, but that's fine of copurse | |
| 18:42:21 | dansmith | anything that does anything on an instance gets the instance record from the cell in the api | |
| 18:42:28 | sean-k-mooney | well the embeded flavor is aslo stored in the api db in the request spec | |
| 18:42:53 | dansmith | it's stored as it was at the time of creation, but not necessarily the same as what the instance has now | |
| 18:43:04 | dansmith | and if we've done a data migration or something they could have diverged | |
| 18:43:11 | sean-k-mooney | oh that wont be update after reisze? | |
| 18:43:18 | sean-k-mooney | ah ok | |
| 18:43:25 | dansmith | the instance's actual flavor is what we should use, and it's no more expensive to get | |
| 18:43:44 | sean-k-mooney | sure makes sense | |
| 18:45:16 | mriedem | the requestspec.flavor is updated as part of a resize, but .... there be bugs | |
| 18:50:04 | spatel | sean-k-mooney: here is the virsh capability - http://paste.openstack.org/show/789301/ | |
| 18:51:00 | spatel | here is the vCPU pinning map - http://paste.openstack.org/show/789302/ | |
| 18:58:23 | sean-k-mooney | ok so the first 15 cpus are pinned to host numa node 0 | |
| 18:58:33 | spatel | Yes | |
| 18:58:59 | sean-k-mooney | and the second 15 are all pinned to host numa node 1 | |
| 18:59:02 | sean-k-mooney | so this is correct | |
| 18:59:06 | spatel | Yes | |
| 18:59:51 | spatel | if you see all looks correct from VM point of view.. | |
| 19:00:21 | spatel | VM correctly mapped its vCPU pins across physical numa nodes | |
| 19:00:43 | sean-k-mooney | yes so you left out one of the imporatn numa elelmnt form the vm | |
| 19:00:59 | spatel | ? | |
| 19:01:10 | sean-k-mooney | you left out the numa element | |
| 19:01:11 | mriedem | important numa elements from the vm | |
| 19:01:20 | mriedem | translation-as-a-service | |
| 19:01:35 | sean-k-mooney | oh ya i miss typed talking downstream as well | |
| 19:01:36 | spatel | I am not following you guys.. | |
| 19:01:39 | sean-k-mooney | ill slow down | |
| 19:01:56 | mriedem | remember how this isn't a support channel? | |
| 19:02:30 | sean-k-mooney | nova will map the first half of the cpus to numa node 0 in the guess so cores 0-15 and form what you have show it is working correctly | |
| 19:03:05 | spatel | Yes.. that is what i am also saying.. CPU pins correctly mapped out to NUMA nodes | |
| 19:03:52 | spatel | This is what NUMA looks inside VM0 -> http://paste.openstack.org/show/789304/ | |
| 19:04:21 | spatel | now question is when i run application why i am seeing poor performance? | |
| 19:04:50 | Sundar | gibi: Any pointers to how you handled reschedule for bandwidth provider? | |
| 19:04:56 | spatel | Do you think i need to do something with -> hw:cpu_thread_policy ? | |
| 19:05:28 | sean-k-mooney | spatel: it woudl appear the aplication is not correctly optimising for the numa toplogy. | |
| 19:05:41 | sean-k-mooney | hw:cpu_thread_polciy wont help | |
| 19:06:23 | sean-k-mooney | you can alther the cpu thread and socket toplogy in the guest which might but this does not seeam to be a nova/libvirt issue | |
| 19:07:12 | sean-k-mooney | spatel: my guess is that you have not toled the guest it is using hyper treads so it is miss allocationg threads internally to cores it think are independed but are not | |
| 19:07:33 | spatel | I am just trying to understand if i run application on bare metal then performance is great but when i create VM with all CPU then performance is bad.. | |
| 19:07:36 | sean-k-mooney | hw:cpu_thread_policy=isolate woudl help but it woudl reduce the available core count | |
| 19:08:06 | sean-k-mooney | you shoudl set hw:cpu_treads=2 and hw:cpu_sockets=2 | |
| 19:08:11 | spatel | how may core count it will reduce ? | |
| 19:08:16 | sean-k-mooney | but you cant do that with 30 cpus | |
| 19:08:26 | sean-k-mooney | you need either 28 or 32 | |
| 19:08:35 | spatel | i will go with 28.. | |
| 19:08:39 | sean-k-mooney | so that you have an even number of cpus per numa node/socket | |
| 19:08:39 | Sundar | gibi: "does the series gracefully handle that move operations are not supported yet with accelerators?" There was an earlier attempt to explicitly block unsupported ops: https://review.opendev.org/#/c/674726/ But it was decided that it is not needed, and we should document the supported ops in Cyborg. | |
| 19:09:20 | spatel | you are saying i should set flavor -> hw:cpu_treads=2 and hw:cpu_sockets=2 and run benchmark right? | |
| 19:09:35 | sean-k-mooney | spatel: set hw:cpu_treads=2 and hw:cpu_sockets=2 and do not set hw:cpu_thread_policy=isolate and see if that fixes the issue | |
| 19:09:48 | spatel | sean-k-mooney: sounds good | |
| 19:10:12 | sean-k-mooney | spatel: actuly i misspelled the first one check the flavor docs to confrim they are right | |
| 19:10:27 | sean-k-mooney | spatel: https://docs.openstack.org/nova/latest/user/flavors.html | |
| 19:10:37 | spatel | I will check doc don't worry all i need clue to try something :) | |
| 19:11:17 | Sundar | gibi: "nova needs to grow support for migration with arqs" Since Cyborg does PCI passthrough, live migration is not supported. | |
| 19:11:40 | spatel | sean-k-mooney: you are saying that option will tell my VM you have two socket.. | |
| 19:11:41 | sean-k-mooney | by default openstack emulates each gust cpu as a sperate socket which is incorrect if you have HT enabled on the host | |
| 19:11:54 | spatel | I do have HT enabled | |
| 19:12:17 | sean-k-mooney | yep so you want to tell the vm it has 2 threads per cpu and in this case 2 socket 1 per numa node | |
| 19:12:34 | sean-k-mooney | that will help the guest kerenl make correct schduling decisions | |
| 19:13:03 | spatel | sean-k-mooney: i think you are correct.. that could be the issue why erlang getting confused | |
| 19:13:19 | spatel | because erlang also run inside VM and it should have own scheduler | |
| 19:14:40 | spatel | sean-k-mooney: I will keep you posted about my testing.. (today isn't possible but Monday i will have some result. | |
| 19:15:44 | spatel | sean-k-mooney: let me create i should set hw:cpu_treads=2 and hw:cpu_sockets=2 and hw:numa_node=2 right? | |
| 19:16:06 | spatel | otherwise it won't let me run my VM on two numa nodes | |
| 19:16:52 | sean-k-mooney | the socket and numa nodes dont have to match but it general works better | |
| 19:17:13 | Sundar | sean-k-mooney: Catching up on earlier discussion about instance ops with Cyborg. Explicit blocking of requests with a device profile name in the extra specs ( https://review.opendev.org/#/c/674726/) was not considered the way to go. Is that still your thinking? | |
| 19:18:11 | spatel | sean-k-mooney: i will set all three option and give it a try.. thank you.. | |
| 19:18:38 | sean-k-mooney | Sundar: am honestly i dont rember the full context of the discussin. i know we discussed this at lenght but cant recal what the decision was | |