| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-07 | |||
| 17:44:39 | gibi | I saw that two of you already handling this seris so I decided to focus energies elsewhere. but agree that fresh eyes helps to se things differently | |
| 17:48:16 | dansmith | yup | |
| 18:10:41 | spatel | sean-k-mooney: I am back now, sorry was in back to back meeting | |
| 18:38:05 | sean-k-mooney | dansmith: we have a check that blocks live migration if there are pci_request spec object that are not related to neutorn sriov ports | |
| 18:38:18 | sean-k-mooney | dansmith: but that wont block cyborg devices | |
| 18:38:25 | dansmith | sean-k-mooney: okay so something similar for cyborg I guess | |
| 18:38:31 | sean-k-mooney | since we also dont have pci_request spec objects | |
| 18:39:04 | sean-k-mooney | well for now we could just check if the flavor has accl:device-profile or whatever teh extraspec is | |
| 18:39:18 | dansmith | right | |
| 18:40:22 | sean-k-mooney | im not sure we will be able to do that check in the api because we would need to check the embeded flaovr form the cell db. actully no we can. we can grab it from the request_spec in the api db | |
| 18:40:42 | sean-k-mooney | so ya we can reject the live migration in the api layer until we support that | |
| 18:41:07 | dansmith | not sure what the problem is.. the api can look at the instance's embedded flavor | |
| 18:41:30 | sean-k-mooney | yep it can so no problem | |
| 18:41:45 | sean-k-mooney | i was thinking we might need to do a down call to the cell db to get it but we dont | |
| 18:41:49 | sean-k-mooney | so all good | |
| 18:42:06 | dansmith | we do to get the actual embedded flavor, but that's fine of copurse | |
| 18:42:21 | dansmith | anything that does anything on an instance gets the instance record from the cell in the api | |
| 18:42:28 | sean-k-mooney | well the embeded flavor is aslo stored in the api db in the request spec | |
| 18:42:53 | dansmith | it's stored as it was at the time of creation, but not necessarily the same as what the instance has now | |
| 18:43:04 | dansmith | and if we've done a data migration or something they could have diverged | |
| 18:43:11 | sean-k-mooney | oh that wont be update after reisze? | |
| 18:43:18 | sean-k-mooney | ah ok | |
| 18:43:25 | dansmith | the instance's actual flavor is what we should use, and it's no more expensive to get | |
| 18:43:44 | sean-k-mooney | sure makes sense | |
| 18:45:16 | mriedem | the requestspec.flavor is updated as part of a resize, but .... there be bugs | |
| 18:50:04 | spatel | sean-k-mooney: here is the virsh capability - http://paste.openstack.org/show/789301/ | |
| 18:51:00 | spatel | here is the vCPU pinning map - http://paste.openstack.org/show/789302/ | |
| 18:58:23 | sean-k-mooney | ok so the first 15 cpus are pinned to host numa node 0 | |
| 18:58:33 | spatel | Yes | |
| 18:58:59 | sean-k-mooney | and the second 15 are all pinned to host numa node 1 | |
| 18:59:02 | sean-k-mooney | so this is correct | |
| 18:59:06 | spatel | Yes | |
| 18:59:51 | spatel | if you see all looks correct from VM point of view.. | |
| 19:00:21 | spatel | VM correctly mapped its vCPU pins across physical numa nodes | |
| 19:00:43 | sean-k-mooney | yes so you left out one of the imporatn numa elelmnt form the vm | |
| 19:00:59 | spatel | ? | |
| 19:01:10 | sean-k-mooney | you left out the numa element | |
| 19:01:11 | mriedem | important numa elements from the vm | |
| 19:01:20 | mriedem | translation-as-a-service | |
| 19:01:35 | sean-k-mooney | oh ya i miss typed talking downstream as well | |
| 19:01:36 | spatel | I am not following you guys.. | |
| 19:01:39 | sean-k-mooney | ill slow down | |
| 19:01:56 | mriedem | remember how this isn't a support channel? | |
| 19:02:30 | sean-k-mooney | nova will map the first half of the cpus to numa node 0 in the guess so cores 0-15 and form what you have show it is working correctly | |
| 19:03:05 | spatel | Yes.. that is what i am also saying.. CPU pins correctly mapped out to NUMA nodes | |
| 19:03:52 | spatel | This is what NUMA looks inside VM0 -> http://paste.openstack.org/show/789304/ | |
| 19:04:21 | spatel | now question is when i run application why i am seeing poor performance? | |
| 19:04:50 | Sundar | gibi: Any pointers to how you handled reschedule for bandwidth provider? | |
| 19:04:56 | spatel | Do you think i need to do something with -> hw:cpu_thread_policy ? | |
| 19:05:28 | sean-k-mooney | spatel: it woudl appear the aplication is not correctly optimising for the numa toplogy. | |
| 19:05:41 | sean-k-mooney | hw:cpu_thread_polciy wont help | |
| 19:06:23 | sean-k-mooney | you can alther the cpu thread and socket toplogy in the guest which might but this does not seeam to be a nova/libvirt issue | |
| 19:07:12 | sean-k-mooney | spatel: my guess is that you have not toled the guest it is using hyper treads so it is miss allocationg threads internally to cores it think are independed but are not | |
| 19:07:33 | spatel | I am just trying to understand if i run application on bare metal then performance is great but when i create VM with all CPU then performance is bad.. | |
| 19:07:36 | sean-k-mooney | hw:cpu_thread_policy=isolate woudl help but it woudl reduce the available core count | |
| 19:08:06 | sean-k-mooney | you shoudl set hw:cpu_treads=2 and hw:cpu_sockets=2 | |
| 19:08:11 | spatel | how may core count it will reduce ? | |
| 19:08:16 | sean-k-mooney | but you cant do that with 30 cpus | |
| 19:08:26 | sean-k-mooney | you need either 28 or 32 | |
| 19:08:35 | spatel | i will go with 28.. | |
| 19:08:39 | sean-k-mooney | so that you have an even number of cpus per numa node/socket | |
| 19:08:39 | Sundar | gibi: "does the series gracefully handle that move operations are not supported yet with accelerators?" There was an earlier attempt to explicitly block unsupported ops: https://review.opendev.org/#/c/674726/ But it was decided that it is not needed, and we should document the supported ops in Cyborg. | |
| 19:09:20 | spatel | you are saying i should set flavor -> hw:cpu_treads=2 and hw:cpu_sockets=2 and run benchmark right? | |
| 19:09:35 | sean-k-mooney | spatel: set hw:cpu_treads=2 and hw:cpu_sockets=2 and do not set hw:cpu_thread_policy=isolate and see if that fixes the issue | |
| 19:09:48 | spatel | sean-k-mooney: sounds good | |
| 19:10:12 | sean-k-mooney | spatel: actuly i misspelled the first one check the flavor docs to confrim they are right | |
| 19:10:27 | sean-k-mooney | spatel: https://docs.openstack.org/nova/latest/user/flavors.html | |
| 19:10:37 | spatel | I will check doc don't worry all i need clue to try something :) | |
| 19:11:17 | Sundar | gibi: "nova needs to grow support for migration with arqs" Since Cyborg does PCI passthrough, live migration is not supported. | |
| 19:11:40 | spatel | sean-k-mooney: you are saying that option will tell my VM you have two socket.. | |
| 19:11:41 | sean-k-mooney | by default openstack emulates each gust cpu as a sperate socket which is incorrect if you have HT enabled on the host | |
| 19:11:54 | spatel | I do have HT enabled | |
| 19:12:17 | sean-k-mooney | yep so you want to tell the vm it has 2 threads per cpu and in this case 2 socket 1 per numa node | |
| 19:12:34 | sean-k-mooney | that will help the guest kerenl make correct schduling decisions | |
| 19:13:03 | spatel | sean-k-mooney: i think you are correct.. that could be the issue why erlang getting confused | |
| 19:13:19 | spatel | because erlang also run inside VM and it should have own scheduler | |
| 19:14:40 | spatel | sean-k-mooney: I will keep you posted about my testing.. (today isn't possible but Monday i will have some result. | |
| 19:15:44 | spatel | sean-k-mooney: let me create i should set hw:cpu_treads=2 and hw:cpu_sockets=2 and hw:numa_node=2 right? | |
| 19:16:06 | spatel | otherwise it won't let me run my VM on two numa nodes | |
| 19:16:52 | sean-k-mooney | the socket and numa nodes dont have to match but it general works better | |
| 19:17:13 | Sundar | sean-k-mooney: Catching up on earlier discussion about instance ops with Cyborg. Explicit blocking of requests with a device profile name in the extra specs ( https://review.opendev.org/#/c/674726/) was not considered the way to go. Is that still your thinking? | |
| 19:18:11 | spatel | sean-k-mooney: i will set all three option and give it a try.. thank you.. | |
| 19:18:38 | sean-k-mooney | Sundar: am honestly i dont rember the full context of the discussin. i know we discussed this at lenght but cant recal what the decision was | |
| 19:19:32 | sean-k-mooney | i think we said skip the explcit check but document what works? but we can aslo block it if we want | |
| 19:20:18 | sean-k-mooney | So the consensus is Option 2: | |
| 19:20:19 | sean-k-mooney | > Gradually phase in the support for the server operations and document the limitations in the meantime but don't actively block them in the API like this change does. | |
| 19:20:21 | sean-k-mooney | We will state the limitations, if any, in Cyborg documentation. | |
| 19:20:30 | sean-k-mooney | that was the last comment on that | |
| 19:22:11 | sean-k-mooney | option 2 was "2. Gradually phase in the support for the server operations and document the limitations in the meantime but don't actively block them in the API like this change does. They either work (by chance) or they don't, but they aren't officially supported. Once they are supported, we patch them in without a new microversion as bug fixes (or just claim test support so they are no longer | |
| 19:22:13 | sean-k-mooney | considered experimental)." | |
| 19:23:18 | sean-k-mooney | gibi: dansmith ^ are ye still ok with that regarding the livemigation check | |
| 19:23:44 | sean-k-mooney | so no check for now and document. and we can add one if we want in the future | |
| 19:25:37 | dansmith | sean-k-mooney: where was that? in the spec? | |
| 19:25:46 | dansmith | it really depends on what the result is | |
| 19:25:55 | sean-k-mooney | https://review.opendev.org/#/c/674726/ | |
| 19:26:01 | sean-k-mooney | its was a nova patch | |
| 19:26:11 | dansmith | if it's data corruption, state intervention required, etc then it needs a check to be graceful | |
| 19:26:22 | dansmith | if it fails in some reasonable way then I'm not so concerned | |
| 19:26:42 | sean-k-mooney | dansmith: i think libvirt will raise an error | |
| 19:27:03 | sean-k-mooney | qemu will reject a migration if the domain has a hostdev that is not of type usb | |