| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 18:42:25 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-n-cpu.txt.gz?severity=4 | |
| 18:42:44 | sean-k-mooney | ah let me get the link | |
| 18:42:53 | sean-k-mooney | https://review.opendev.org/#/c/681771/1 | |
| 18:43:29 | dansmith | there's some failed db lookups | |
| 18:43:36 | dansmith | like upcalls or something? I don't even know | |
| 18:43:46 | dansmith | Sep 12 18:10:31.308872 ubuntu-bionic-expanded-fortnebula-regionone-0011217193 nova-compute[14200]: ERROR nova.compute.manager [instance: 44f764db-e0ef-492a-8ac4-c9b110037b7d] oslo_messaging.rpc.client.RemoteError: Remote error: CantStartEngineError No sql_connection parameter is established | |
| 18:44:01 | dansmith | (44f764db-e0ef-492a-8ac4-c9b110037b7d) to current host (ubuntu-bionic-expanded-fortnebula-regionone-0011217193). | |
| 18:44:01 | dansmith | Sep 12 18:10:31.376947 ubuntu-bionic-expanded-fortnebula-regionone-0011217193 nova-compute[14200]: INFO nova.compute.manager [None req-a4e37fd4-9b29-472f-82f8-4cc5d6ba78cd tempest-TestNetworkAdvancedServerOps-353388395 tempest-TestNetworkAdvancedServerOps-353388395] [instance: 44f764db-e0ef-492a-8ac4-c9b110037b7d] Setting instance back to active after: Instance rollback performed due to: Unable to migrate instance | |
| 18:44:32 | sean-k-mooney | ya i think i have seen those before | |
| 18:44:44 | sean-k-mooney | but i dont know where | |
| 18:44:54 | artom | Wasn't that happening when you were testing just my patches? | |
| 18:45:19 | sean-k-mooney | perhaps | |
| 18:45:28 | sean-k-mooney | we shoudl have the ci logs to check | |
| 18:45:39 | artom | Yeah, trying to find them | |
| 18:45:42 | sean-k-mooney | i think i have seen that on master before too | |
| 18:45:56 | dansmith | it's trying to do a reschedule, | |
| 18:45:57 | sean-k-mooney | logstsh shoudl tell us i guess | |
| 18:46:05 | dansmith | but I'm guessing that means we failed to do something we should have been able to do, | |
| 18:46:14 | dansmith | like claiming pcpus again or something | |
| 18:47:00 | sean-k-mooney | so we have this error https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-n-cpu.txt.gz#4825 | |
| 18:47:17 | sean-k-mooney | that is what i expect to see wehn we live migrate without updating things properly | |
| 18:47:31 | dansmith | the logs I saw were cold migrate I think | |
| 18:48:04 | dansmith | yep, resize | |
| 18:54:06 | sean-k-mooney | inventory data: {'VCPU': {'total': 7, 'reserved': 0, 'min_unit': 1, 'max_unit': 7, 'step_size': 1, 'allocation_ratio': 16.0}, | |
| 18:54:19 | sean-k-mooney | so that isnt workign correctly right | |
| 18:54:46 | sean-k-mooney | this should be the combiantion fo stephens change + artoms | |
| 18:55:07 | sean-k-mooney | so i would expect that to contain PCPUS | |
| 18:55:22 | sean-k-mooney | i have other jobs that are just stephens code running too | |
| 18:55:55 | dansmith | wouldn't we fail to boot at all if there wasn't any inventory? | |
| 18:55:59 | dansmith | meaning, fail to get to the compute at all | |
| 18:56:55 | sean-k-mooney | i think its reporting inventory of PCPU and either not traslating or doing the fallback | |
| 18:57:06 | artom | I didn't touch any of the scheduler stuff in my code - I assumed it handled placement allocations correctly for a live migration | |
| 18:57:25 | artom | Maybe that part needs to be filled in for PCPUs? | |
| 18:58:10 | dansmith | inventory data: {'VCPU': {'total': 7, 'reserved': 0, 'min_unit': 1, 'max_unit': 7, 'step_size': 1, 'allocation_ratio': 16.0}, 'MEMORY_MB': {'total': 16039, 'reserved': 512, 'min_unit': 1, 'max_unit': 16039, 'step_size': 1, 'allocation_ratio': 1.5}, 'DISK_GB': {'total': 74, 'reserved': 0, 'min_unit': 1, 'max_unit': 74, 'step_size': 1, 'allocation_ratio': 1.0} | |
| 18:58:14 | dansmith | am I missing the PCPU in there? | |
| 18:59:18 | dansmith | Translating request for VCPU=2 to VCPU=0,PCPU=2 | |
| 18:59:44 | sean-k-mooney | no | |
| 18:59:44 | dansmith | I thought that was supposed to happen in the scheduler request, but that's in the compute log | |
| 19:00:06 | sean-k-mooney | so did i | |
| 19:00:38 | sean-k-mooney | could it happen in the compute on resize maybe | |
| 19:00:46 | dansmith | no, it can't call the scheduler | |
| 19:00:59 | sean-k-mooney | well its in the scheduler utils | |
| 19:01:05 | sean-k-mooney | but ya | |
| 19:01:13 | dansmith | unless that's just being logged from a util method we're calling to generate an allocation update or something | |
| 19:01:19 | sean-k-mooney | do we have placmenet logs | |
| 19:02:43 | sean-k-mooney | that is a request for PCPUs right https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-placement-api.txt.gz#342 | |
| 19:03:13 | sean-k-mooney | its using the fallback | |
| 19:03:37 | dansmith | that's a req from the scheduler I assume, | |
| 19:03:46 | dansmith | I'm not sure why we're seeing the translation message in the compute log tho | |
| 19:04:18 | sean-k-mooney | yes and just below it we see it request again with VCPUS | |
| 19:05:22 | sean-k-mooney | stephenfin: did i miss adding something in the nova conf | |
| 19:05:33 | stephenfin | sean-k-mooney: what did you add? | |
| 19:05:55 | sean-k-mooney | just vcpu_pin_set in this case + cpu_shared_set | |
| 19:06:07 | sean-k-mooney | but it hsould still get PCPU no? | |
| 19:06:14 | stephenfin | nope | |
| 19:06:36 | dansmith | stephenfin: do you know why we're logging that translation thing in the compute log? is it from generating allocations to compare in the periodic or something? | |
| 19:06:36 | sean-k-mooney | oh ok | |
| 19:06:37 | stephenfin | the host won't do reshape or report PCPU until '[compute] cpu_dedicated_set' is configured. That's our switchover | |
| 19:06:54 | sean-k-mooney | ah ok | |
| 19:07:03 | sean-k-mooney | ill update that job then | |
| 19:07:12 | stephenfin | dansmith: which one? | |
| 19:07:15 | sean-k-mooney | the other use cpu_share_set and cpu_dedicated_set | |
| 19:07:20 | stephenfin | oh Translating request for VCPU=2 to VCPU=0,PCPU=2 | |
| 19:07:48 | stephenfin | yeah, sec | |
| 19:07:48 | dansmith | yeah | |
| 19:08:14 | dansmith | I saw that, forgot it was in the compute log and was like "cool, cool, we're ... wait a sec." :) | |
| 19:09:16 | stephenfin | so this is the code that does the translation https://review.opendev.org/#/c/671801/48/nova/scheduler/utils.py | |
| 19:09:34 | stephenfin | and the question is where do we build ResourceRequest object on the compute ndoe | |
| 19:09:36 | stephenfin | *node | |
| 19:09:43 | stephenfin | because I didn't add that :) | |
| 19:09:54 | dansmith | if we're legit logging that we should probably change that wording a little | |
| 19:10:32 | stephenfin | got it | |
| 19:10:45 | stephenfin | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L595-L602 | |
| 19:10:50 | sean-k-mooney | "cpu_dedicated_set: 0-6" is the correct spelling yes | |
| 19:11:02 | stephenfin | yup, and it has to be in the compute group | |
| 19:11:12 | sean-k-mooney | ya | |
| 19:12:10 | stephenfin | dansmith: That's from the vCPU model selection code and it was added as a quick way to extract the traits | |
| 19:12:54 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] numa + pcpus in placment live migration tests https://review.opendev.org/681771 | |
| 19:13:04 | stephenfin | So agreed, it's misleading, and I didn't see it before because the vCPU model selection change only merged last night | |
| 19:13:43 | dansmith | s'cool, just sayin' | |
| 19:13:56 | sean-k-mooney | ok that re running | |
| 19:14:35 | sean-k-mooney | the 3 jobs that are just running the PCPU code shoudl report back in the next 20 mins or so the last one with numa will take about an hour and a half | |
| 19:14:45 | stephenfin | cool :) | |
| 19:15:04 | stephenfin | The solution, fwiw (and IMO), is what efried has asked for elsewhere: a util function to pull traits and resources from a flavor in a standard way, so we don't have a load of 'resourceNN:<type>' regexes about the place | |
| 19:15:14 | stephenfin | regexes? regexi? | |
| 19:15:33 | dansmith | sean-k-mooney: so what's the deal on the failed db queries? | |
| 19:15:50 | dansmith | sean-k-mooney: worried that maybe we're lazy-loading some api-only field or something | |
| 19:16:10 | sean-k-mooney | if they dont show up the ones that is stephens only then its from artoms series | |
| 19:16:16 | sean-k-mooney | if its in both its from master | |
| 19:17:24 | sean-k-mooney | this https://review.opendev.org/#/c/681771/1 is the combiend job and https://review.opendev.org/#/c/681807/2 is the 3 pcpu only jobs | |
| 19:17:50 | sean-k-mooney | i think i did see that with artoms series but not in all versions | |
| 19:18:22 | sean-k-mooney | could it be the late upcall for anti affintiy | |
| 19:19:02 | sean-k-mooney | if we are lazy loadign somethign im not sure what it would be off the top of my head | |
| 19:20:30 | stephenfin | sean-k-mooney, dansmith: WIP manual testing notes here too, btw https://etherpad.openstack.org/p/nova-cpu-resources | |
| 19:21:19 | stephenfin | I'm at controller+compute updated to use PCPU code with other compute using plain master (without PCPU code). Onto setting 'cpu_dedicated_set' on the former now | |
| 19:21:56 | stephenfin | I went with master because DevStack wouldn't deploy from stable/stein and I figured this would be easier simulate "upgrades" from nova's perspective | |
| 19:22:08 | sean-k-mooney | i ment to get food earilar and still havent so im gong to run for an 30 mins to an hour | |
| 19:22:38 | stephenfin | I just want to test the reshaper and create/move operations with nodes with and without the PCPU stuff, after all | |
| 19:22:56 | dansmith | sean-k-mooney: I dunno how it could be unrelated if it works on master (the lazy load) | |
| 19:26:20 | sean-k-mooney | its likely an issue in artoms code then if stephen has not seen it in his testing? | |