| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 18:42:44 | sean-k-mooney | ah let me get the link | |
| 18:42:53 | sean-k-mooney | https://review.opendev.org/#/c/681771/1 | |
| 18:43:29 | dansmith | there's some failed db lookups | |
| 18:43:36 | dansmith | like upcalls or something? I don't even know | |
| 18:43:46 | dansmith | Sep 12 18:10:31.308872 ubuntu-bionic-expanded-fortnebula-regionone-0011217193 nova-compute[14200]: ERROR nova.compute.manager [instance: 44f764db-e0ef-492a-8ac4-c9b110037b7d] oslo_messaging.rpc.client.RemoteError: Remote error: CantStartEngineError No sql_connection parameter is established | |
| 18:44:01 | dansmith | Sep 12 18:10:31.376947 ubuntu-bionic-expanded-fortnebula-regionone-0011217193 nova-compute[14200]: INFO nova.compute.manager [None req-a4e37fd4-9b29-472f-82f8-4cc5d6ba78cd tempest-TestNetworkAdvancedServerOps-353388395 tempest-TestNetworkAdvancedServerOps-353388395] [instance: 44f764db-e0ef-492a-8ac4-c9b110037b7d] Setting instance back to active after: Instance rollback performed due to: Unable to migrate instance | |
| 18:44:01 | dansmith | (44f764db-e0ef-492a-8ac4-c9b110037b7d) to current host (ubuntu-bionic-expanded-fortnebula-regionone-0011217193). | |
| 18:44:32 | sean-k-mooney | ya i think i have seen those before | |
| 18:44:44 | sean-k-mooney | but i dont know where | |
| 18:44:54 | artom | Wasn't that happening when you were testing just my patches? | |
| 18:45:19 | sean-k-mooney | perhaps | |
| 18:45:28 | sean-k-mooney | we shoudl have the ci logs to check | |
| 18:45:39 | artom | Yeah, trying to find them | |
| 18:45:42 | sean-k-mooney | i think i have seen that on master before too | |
| 18:45:56 | dansmith | it's trying to do a reschedule, | |
| 18:45:57 | sean-k-mooney | logstsh shoudl tell us i guess | |
| 18:46:05 | dansmith | but I'm guessing that means we failed to do something we should have been able to do, | |
| 18:46:14 | dansmith | like claiming pcpus again or something | |
| 18:47:00 | sean-k-mooney | so we have this error https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-n-cpu.txt.gz#4825 | |
| 18:47:17 | sean-k-mooney | that is what i expect to see wehn we live migrate without updating things properly | |
| 18:47:31 | dansmith | the logs I saw were cold migrate I think | |
| 18:48:04 | dansmith | yep, resize | |
| 18:54:06 | sean-k-mooney | inventory data: {'VCPU': {'total': 7, 'reserved': 0, 'min_unit': 1, 'max_unit': 7, 'step_size': 1, 'allocation_ratio': 16.0}, | |
| 18:54:19 | sean-k-mooney | so that isnt workign correctly right | |
| 18:54:46 | sean-k-mooney | this should be the combiantion fo stephens change + artoms | |
| 18:55:07 | sean-k-mooney | so i would expect that to contain PCPUS | |
| 18:55:22 | sean-k-mooney | i have other jobs that are just stephens code running too | |
| 18:55:55 | dansmith | wouldn't we fail to boot at all if there wasn't any inventory? | |
| 18:55:59 | dansmith | meaning, fail to get to the compute at all | |
| 18:56:55 | sean-k-mooney | i think its reporting inventory of PCPU and either not traslating or doing the fallback | |
| 18:57:06 | artom | I didn't touch any of the scheduler stuff in my code - I assumed it handled placement allocations correctly for a live migration | |
| 18:57:25 | artom | Maybe that part needs to be filled in for PCPUs? | |
| 18:58:10 | dansmith | inventory data: {'VCPU': {'total': 7, 'reserved': 0, 'min_unit': 1, 'max_unit': 7, 'step_size': 1, 'allocation_ratio': 16.0}, 'MEMORY_MB': {'total': 16039, 'reserved': 512, 'min_unit': 1, 'max_unit': 16039, 'step_size': 1, 'allocation_ratio': 1.5}, 'DISK_GB': {'total': 74, 'reserved': 0, 'min_unit': 1, 'max_unit': 74, 'step_size': 1, 'allocation_ratio': 1.0} | |
| 18:58:14 | dansmith | am I missing the PCPU in there? | |
| 18:59:18 | dansmith | Translating request for VCPU=2 to VCPU=0,PCPU=2 | |
| 18:59:44 | dansmith | I thought that was supposed to happen in the scheduler request, but that's in the compute log | |
| 18:59:44 | sean-k-mooney | no | |
| 19:00:06 | sean-k-mooney | so did i | |
| 19:00:38 | sean-k-mooney | could it happen in the compute on resize maybe | |
| 19:00:46 | dansmith | no, it can't call the scheduler | |
| 19:00:59 | sean-k-mooney | well its in the scheduler utils | |
| 19:01:05 | sean-k-mooney | but ya | |
| 19:01:13 | dansmith | unless that's just being logged from a util method we're calling to generate an allocation update or something | |
| 19:01:19 | sean-k-mooney | do we have placmenet logs | |
| 19:02:43 | sean-k-mooney | that is a request for PCPUs right https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-placement-api.txt.gz#342 | |
| 19:03:13 | sean-k-mooney | its using the fallback | |
| 19:03:37 | dansmith | that's a req from the scheduler I assume, | |
| 19:03:46 | dansmith | I'm not sure why we're seeing the translation message in the compute log tho | |
| 19:04:18 | sean-k-mooney | yes and just below it we see it request again with VCPUS | |
| 19:05:22 | sean-k-mooney | stephenfin: did i miss adding something in the nova conf | |
| 19:05:33 | stephenfin | sean-k-mooney: what did you add? | |
| 19:05:55 | sean-k-mooney | just vcpu_pin_set in this case + cpu_shared_set | |
| 19:06:07 | sean-k-mooney | but it hsould still get PCPU no? | |
| 19:06:14 | stephenfin | nope | |
| 19:06:36 | sean-k-mooney | oh ok | |
| 19:06:36 | dansmith | stephenfin: do you know why we're logging that translation thing in the compute log? is it from generating allocations to compare in the periodic or something? | |
| 19:06:37 | stephenfin | the host won't do reshape or report PCPU until '[compute] cpu_dedicated_set' is configured. That's our switchover | |
| 19:06:54 | sean-k-mooney | ah ok | |
| 19:07:03 | sean-k-mooney | ill update that job then | |
| 19:07:12 | stephenfin | dansmith: which one? | |
| 19:07:15 | sean-k-mooney | the other use cpu_share_set and cpu_dedicated_set | |
| 19:07:20 | stephenfin | oh Translating request for VCPU=2 to VCPU=0,PCPU=2 | |
| 19:07:48 | dansmith | yeah | |
| 19:07:48 | stephenfin | yeah, sec | |
| 19:08:14 | dansmith | I saw that, forgot it was in the compute log and was like "cool, cool, we're ... wait a sec." :) | |
| 19:09:16 | stephenfin | so this is the code that does the translation https://review.opendev.org/#/c/671801/48/nova/scheduler/utils.py | |
| 19:09:34 | stephenfin | and the question is where do we build ResourceRequest object on the compute ndoe | |
| 19:09:36 | stephenfin | *node | |
| 19:09:43 | stephenfin | because I didn't add that :) | |
| 19:09:54 | dansmith | if we're legit logging that we should probably change that wording a little | |
| 19:10:32 | stephenfin | got it | |
| 19:10:45 | stephenfin | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L595-L602 | |
| 19:10:50 | sean-k-mooney | "cpu_dedicated_set: 0-6" is the correct spelling yes | |
| 19:11:02 | stephenfin | yup, and it has to be in the compute group | |
| 19:11:12 | sean-k-mooney | ya | |
| 19:12:10 | stephenfin | dansmith: That's from the vCPU model selection code and it was added as a quick way to extract the traits | |
| 19:12:54 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] numa + pcpus in placment live migration tests https://review.opendev.org/681771 | |
| 19:13:04 | stephenfin | So agreed, it's misleading, and I didn't see it before because the vCPU model selection change only merged last night | |
| 19:13:43 | dansmith | s'cool, just sayin' | |
| 19:13:56 | sean-k-mooney | ok that re running | |
| 19:14:35 | sean-k-mooney | the 3 jobs that are just running the PCPU code shoudl report back in the next 20 mins or so the last one with numa will take about an hour and a half | |
| 19:14:45 | stephenfin | cool :) | |
| 19:15:04 | stephenfin | The solution, fwiw (and IMO), is what efried has asked for elsewhere: a util function to pull traits and resources from a flavor in a standard way, so we don't have a load of 'resourceNN:<type>' regexes about the place | |
| 19:15:14 | stephenfin | regexes? regexi? | |
| 19:15:33 | dansmith | sean-k-mooney: so what's the deal on the failed db queries? | |
| 19:15:50 | dansmith | sean-k-mooney: worried that maybe we're lazy-loading some api-only field or something | |
| 19:16:10 | sean-k-mooney | if they dont show up the ones that is stephens only then its from artoms series | |
| 19:16:16 | sean-k-mooney | if its in both its from master | |
| 19:17:24 | sean-k-mooney | this https://review.opendev.org/#/c/681771/1 is the combiend job and https://review.opendev.org/#/c/681807/2 is the 3 pcpu only jobs | |
| 19:17:50 | sean-k-mooney | i think i did see that with artoms series but not in all versions | |
| 19:18:22 | sean-k-mooney | could it be the late upcall for anti affintiy | |
| 19:19:02 | sean-k-mooney | if we are lazy loadign somethign im not sure what it would be off the top of my head | |
| 19:20:30 | stephenfin | sean-k-mooney, dansmith: WIP manual testing notes here too, btw https://etherpad.openstack.org/p/nova-cpu-resources | |
| 19:21:19 | stephenfin | I'm at controller+compute updated to use PCPU code with other compute using plain master (without PCPU code). Onto setting 'cpu_dedicated_set' on the former now | |
| 19:21:56 | stephenfin | I went with master because DevStack wouldn't deploy from stable/stein and I figured this would be easier simulate "upgrades" from nova's perspective | |
| 19:22:08 | sean-k-mooney | i ment to get food earilar and still havent so im gong to run for an 30 mins to an hour | |
| 19:22:38 | stephenfin | I just want to test the reshaper and create/move operations with nodes with and without the PCPU stuff, after all | |
| 19:22:56 | dansmith | sean-k-mooney: I dunno how it could be unrelated if it works on master (the lazy load) | |
| 19:26:20 | sean-k-mooney | its likely an issue in artoms code then if stephen has not seen it in his testing? | |
| 19:27:02 | dansmith | no, I would think it's more likely that stephen added something that looks at an object field, but hasn't been testing with computes and conductors that don't have access to the api database | |