| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 18:56:55 | sean-k-mooney | i think its reporting inventory of PCPU and either not traslating or doing the fallback | |
| 18:57:06 | artom | I didn't touch any of the scheduler stuff in my code - I assumed it handled placement allocations correctly for a live migration | |
| 18:57:25 | artom | Maybe that part needs to be filled in for PCPUs? | |
| 18:58:10 | dansmith | inventory data: {'VCPU': {'total': 7, 'reserved': 0, 'min_unit': 1, 'max_unit': 7, 'step_size': 1, 'allocation_ratio': 16.0}, 'MEMORY_MB': {'total': 16039, 'reserved': 512, 'min_unit': 1, 'max_unit': 16039, 'step_size': 1, 'allocation_ratio': 1.5}, 'DISK_GB': {'total': 74, 'reserved': 0, 'min_unit': 1, 'max_unit': 74, 'step_size': 1, 'allocation_ratio': 1.0} | |
| 18:58:14 | dansmith | am I missing the PCPU in there? | |
| 18:59:18 | dansmith | Translating request for VCPU=2 to VCPU=0,PCPU=2 | |
| 18:59:44 | sean-k-mooney | no | |
| 18:59:44 | dansmith | I thought that was supposed to happen in the scheduler request, but that's in the compute log | |
| 19:00:06 | sean-k-mooney | so did i | |
| 19:00:38 | sean-k-mooney | could it happen in the compute on resize maybe | |
| 19:00:46 | dansmith | no, it can't call the scheduler | |
| 19:00:59 | sean-k-mooney | well its in the scheduler utils | |
| 19:01:05 | sean-k-mooney | but ya | |
| 19:01:13 | dansmith | unless that's just being logged from a util method we're calling to generate an allocation update or something | |
| 19:01:19 | sean-k-mooney | do we have placmenet logs | |
| 19:02:43 | sean-k-mooney | that is a request for PCPUs right https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-placement-api.txt.gz#342 | |
| 19:03:13 | sean-k-mooney | its using the fallback | |
| 19:03:37 | dansmith | that's a req from the scheduler I assume, | |
| 19:03:46 | dansmith | I'm not sure why we're seeing the translation message in the compute log tho | |
| 19:04:18 | sean-k-mooney | yes and just below it we see it request again with VCPUS | |
| 19:05:22 | sean-k-mooney | stephenfin: did i miss adding something in the nova conf | |
| 19:05:33 | stephenfin | sean-k-mooney: what did you add? | |
| 19:05:55 | sean-k-mooney | just vcpu_pin_set in this case + cpu_shared_set | |
| 19:06:07 | sean-k-mooney | but it hsould still get PCPU no? | |
| 19:06:14 | stephenfin | nope | |
| 19:06:36 | dansmith | stephenfin: do you know why we're logging that translation thing in the compute log? is it from generating allocations to compare in the periodic or something? | |
| 19:06:36 | sean-k-mooney | oh ok | |
| 19:06:37 | stephenfin | the host won't do reshape or report PCPU until '[compute] cpu_dedicated_set' is configured. That's our switchover | |
| 19:06:54 | sean-k-mooney | ah ok | |
| 19:07:03 | sean-k-mooney | ill update that job then | |
| 19:07:12 | stephenfin | dansmith: which one? | |
| 19:07:15 | sean-k-mooney | the other use cpu_share_set and cpu_dedicated_set | |
| 19:07:20 | stephenfin | oh Translating request for VCPU=2 to VCPU=0,PCPU=2 | |
| 19:07:48 | stephenfin | yeah, sec | |
| 19:07:48 | dansmith | yeah | |
| 19:08:14 | dansmith | I saw that, forgot it was in the compute log and was like "cool, cool, we're ... wait a sec." :) | |
| 19:09:16 | stephenfin | so this is the code that does the translation https://review.opendev.org/#/c/671801/48/nova/scheduler/utils.py | |
| 19:09:34 | stephenfin | and the question is where do we build ResourceRequest object on the compute ndoe | |
| 19:09:36 | stephenfin | *node | |
| 19:09:43 | stephenfin | because I didn't add that :) | |
| 19:09:54 | dansmith | if we're legit logging that we should probably change that wording a little | |
| 19:10:32 | stephenfin | got it | |
| 19:10:45 | stephenfin | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L595-L602 | |
| 19:10:50 | sean-k-mooney | "cpu_dedicated_set: 0-6" is the correct spelling yes | |
| 19:11:02 | stephenfin | yup, and it has to be in the compute group | |
| 19:11:12 | sean-k-mooney | ya | |
| 19:12:10 | stephenfin | dansmith: That's from the vCPU model selection code and it was added as a quick way to extract the traits | |
| 19:12:54 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] numa + pcpus in placment live migration tests https://review.opendev.org/681771 | |
| 19:13:04 | stephenfin | So agreed, it's misleading, and I didn't see it before because the vCPU model selection change only merged last night | |
| 19:13:43 | dansmith | s'cool, just sayin' | |
| 19:13:56 | sean-k-mooney | ok that re running | |
| 19:14:35 | sean-k-mooney | the 3 jobs that are just running the PCPU code shoudl report back in the next 20 mins or so the last one with numa will take about an hour and a half | |
| 19:14:45 | stephenfin | cool :) | |
| 19:15:04 | stephenfin | The solution, fwiw (and IMO), is what efried has asked for elsewhere: a util function to pull traits and resources from a flavor in a standard way, so we don't have a load of 'resourceNN:<type>' regexes about the place | |
| 19:15:14 | stephenfin | regexes? regexi? | |
| 19:15:33 | dansmith | sean-k-mooney: so what's the deal on the failed db queries? | |
| 19:15:50 | dansmith | sean-k-mooney: worried that maybe we're lazy-loading some api-only field or something | |
| 19:16:10 | sean-k-mooney | if they dont show up the ones that is stephens only then its from artoms series | |
| 19:16:16 | sean-k-mooney | if its in both its from master | |
| 19:17:24 | sean-k-mooney | this https://review.opendev.org/#/c/681771/1 is the combiend job and https://review.opendev.org/#/c/681807/2 is the 3 pcpu only jobs | |
| 19:17:50 | sean-k-mooney | i think i did see that with artoms series but not in all versions | |
| 19:18:22 | sean-k-mooney | could it be the late upcall for anti affintiy | |
| 19:19:02 | sean-k-mooney | if we are lazy loadign somethign im not sure what it would be off the top of my head | |
| 19:20:30 | stephenfin | sean-k-mooney, dansmith: WIP manual testing notes here too, btw https://etherpad.openstack.org/p/nova-cpu-resources | |
| 19:21:19 | stephenfin | I'm at controller+compute updated to use PCPU code with other compute using plain master (without PCPU code). Onto setting 'cpu_dedicated_set' on the former now | |
| 19:21:56 | stephenfin | I went with master because DevStack wouldn't deploy from stable/stein and I figured this would be easier simulate "upgrades" from nova's perspective | |
| 19:22:08 | sean-k-mooney | i ment to get food earilar and still havent so im gong to run for an 30 mins to an hour | |
| 19:22:38 | stephenfin | I just want to test the reshaper and create/move operations with nodes with and without the PCPU stuff, after all | |
| 19:22:56 | dansmith | sean-k-mooney: I dunno how it could be unrelated if it works on master (the lazy load) | |
| 19:26:20 | sean-k-mooney | its likely an issue in artoms code then if stephen has not seen it in his testing? | |
| 19:27:02 | dansmith | no, I would think it's more likely that stephen added something that looks at an object field, but hasn't been testing with computes and conductors that don't have access to the api database | |
| 19:27:39 | dansmith | artom's code eventually passed this right? | |
| 19:29:21 | mnaser | melwitt: thanks (re os-vif) | |
| 19:31:36 | dansmith | ...still not a damn thing in the gate | |
| 19:33:31 | melwitt | the Nova Penalty™ | |
| 19:40:59 | artom | dansmith, stuff that's been +W'ed pre-FF can be rechecked until it merges, right? | |
| 19:41:06 | artom | It's not like "whelp, FF, killall the things" | |
| 19:42:44 | dansmith | ...mkay | |
| 19:43:56 | stephenfin | dansmith: okay, so, here's what I've tested | |
| 19:44:09 | stephenfin | that I can boot pinned and unpinned instances on master | |
| 19:44:33 | stephenfin | that if I "upgrade" a compute node to use the PCPU stuff, the existing instances keep working | |
| 19:44:48 | stephenfin | that I can boot new instances on that upgraded compute node | |
| 19:45:20 | stephenfin | that if I set 'cpu_dedicated_set', all my pinned instance magically transition to having PCPU inventory | |
| 19:46:18 | stephenfin | (that one was tricky because I (boldly) had pinned and unpinned instances on the same host, so I have to pick a value for 'cpu_dedicated_set' that match the pinset of the pinned instances and still leave some aside for 'cpu_shared_set') | |
| 19:47:13 | stephenfin | that I can continue to boot pinned instances after having set that, and they will go to both the host reporting PCPUs and the host without PCPUs | |
| 19:48:05 | stephenfin | and throughout all of that, placement's view of WTF is happening remains consistent | |
| 19:48:13 | stephenfin | *that placement's | |
| 19:49:07 | stephenfin | dansmith: How's that sounding so far? Anything in particular you want me to double back on or try? | |
| 19:50:07 | stephenfin | Happy to tar up the logs from both compute nodes for your perusal too. They're a bit messy because I've made a few (user-driven) mistakes along the way but they should do to quickly grep for ERRORs | |
| 19:50:40 | stephenfin | The main ones of those I've seen are errors from RabbitMQ about trying again in 1 sec, which I seem to always get in DevStack | |
| 19:51:40 | stephenfin | and libvirt b****** when I tried to set 'cpu_dedicated_set' to a range that excluded pinned cores from existing instances | |
| 19:52:42 | stephenfin | That's an issue already with vcpu_pin_set but I thought I'd fixed that with this bugger https://review.opendev.org/#/c/680107/ I've clearly missed something though and will keep investigating | |
| 19:53:15 | stephenfin | Also, there's a bug in the vCPU model selection code | |
| 19:54:47 | stephenfin | Namely, if I have a required trait and it's not a CPU flag, this will contain at least some None entries https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L600 | |
| 19:55:14 | stephenfin | and we have a later check to see if the result from the function is false'y | |
| 19:55:17 | stephenfin | set([None]) is not false'y | |
| 19:55:28 | stephenfin | patch incoming for that | |
| 20:00:02 | sean-k-mooney | stephenfin: here is the results for https://review.opendev.org/#/c/681827/2 https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_5bb/681827/2/experimental/nova-nfv-multinode/5bb9889/testr_results.html.gz | |
| 20:00:25 | sean-k-mooney | that is your code without artoms | |
| 20:00:39 | sean-k-mooney | the only thing that failed were 2 live migration tests | |