| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 18:38:25 | efried | or "instances" | |
| 18:38:26 | aspiers | Anyway, dinner time | |
| 18:38:27 | sean-k-mooney | for the combinined numa migration + pcpus tests | |
| 18:38:47 | artom | sean-k-mooney, "almost" makes me nervous | |
| 18:38:58 | sean-k-mooney | the ones that failed were not live migration | |
| 18:39:02 | sean-k-mooney | those all passed | |
| 18:39:12 | sean-k-mooney | your code is fine | |
| 18:39:15 | efried | I suspect it would confuse the hell out of Chinese operators if "instance" was 例子 everywhere. | |
| 18:39:25 | artom | sean-k-mooney, well ok, but is stephenfin's code fine? | |
| 18:39:47 | sean-k-mooney | it finished 30 seconds ago so im chekcing | |
| 18:39:59 | mriedem_afk | i'll be back before the meeting | |
| 18:40:00 | aspiers | efried: I have no idea, but anyway none of this is related to any of my patches, so I'm washing my hands ;-) | |
| 18:40:08 | sean-k-mooney | one of the 3 failures was failing to delete a vm | |
| 18:40:10 | efried | Yeah, even at the start of this I was just being pedantic, not arguing for actual change | |
| 18:40:42 | aspiers | I would have thought that the i18n team would have received enough complaints by now if it wasn't right | |
| 18:40:53 | aspiers | Unless Horizon changed something recently, I dunno | |
| 18:41:05 | mriedem_afk | the one guy at a university trying to stand up nova + xen just called out in the ML | |
| 18:41:06 | aspiers | I never noticed it with the British spelling before, but maybe it was always there | |
| 18:41:17 | efried | timeouts | |
| 18:41:32 | sean-k-mooney | ok 1 was failure to delete an instance and 2 failrues were trying to migrete to the same host | |
| 18:41:35 | sean-k-mooney | artom: ^ | |
| 18:41:59 | artom | sean-k-mooney, which review is that from? | |
| 18:42:25 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-n-cpu.txt.gz?severity=4 | |
| 18:42:44 | sean-k-mooney | ah let me get the link | |
| 18:42:53 | sean-k-mooney | https://review.opendev.org/#/c/681771/1 | |
| 18:43:29 | dansmith | there's some failed db lookups | |
| 18:43:36 | dansmith | like upcalls or something? I don't even know | |
| 18:43:46 | dansmith | Sep 12 18:10:31.308872 ubuntu-bionic-expanded-fortnebula-regionone-0011217193 nova-compute[14200]: ERROR nova.compute.manager [instance: 44f764db-e0ef-492a-8ac4-c9b110037b7d] oslo_messaging.rpc.client.RemoteError: Remote error: CantStartEngineError No sql_connection parameter is established | |
| 18:44:01 | dansmith | (44f764db-e0ef-492a-8ac4-c9b110037b7d) to current host (ubuntu-bionic-expanded-fortnebula-regionone-0011217193). | |
| 18:44:01 | dansmith | Sep 12 18:10:31.376947 ubuntu-bionic-expanded-fortnebula-regionone-0011217193 nova-compute[14200]: INFO nova.compute.manager [None req-a4e37fd4-9b29-472f-82f8-4cc5d6ba78cd tempest-TestNetworkAdvancedServerOps-353388395 tempest-TestNetworkAdvancedServerOps-353388395] [instance: 44f764db-e0ef-492a-8ac4-c9b110037b7d] Setting instance back to active after: Instance rollback performed due to: Unable to migrate instance | |
| 18:44:32 | sean-k-mooney | ya i think i have seen those before | |
| 18:44:44 | sean-k-mooney | but i dont know where | |
| 18:44:54 | artom | Wasn't that happening when you were testing just my patches? | |
| 18:45:19 | sean-k-mooney | perhaps | |
| 18:45:28 | sean-k-mooney | we shoudl have the ci logs to check | |
| 18:45:39 | artom | Yeah, trying to find them | |
| 18:45:42 | sean-k-mooney | i think i have seen that on master before too | |
| 18:45:56 | dansmith | it's trying to do a reschedule, | |
| 18:45:57 | sean-k-mooney | logstsh shoudl tell us i guess | |
| 18:46:05 | dansmith | but I'm guessing that means we failed to do something we should have been able to do, | |
| 18:46:14 | dansmith | like claiming pcpus again or something | |
| 18:47:00 | sean-k-mooney | so we have this error https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-n-cpu.txt.gz#4825 | |
| 18:47:17 | sean-k-mooney | that is what i expect to see wehn we live migrate without updating things properly | |
| 18:47:31 | dansmith | the logs I saw were cold migrate I think | |
| 18:48:04 | dansmith | yep, resize | |
| 18:54:06 | sean-k-mooney | inventory data: {'VCPU': {'total': 7, 'reserved': 0, 'min_unit': 1, 'max_unit': 7, 'step_size': 1, 'allocation_ratio': 16.0}, | |
| 18:54:19 | sean-k-mooney | so that isnt workign correctly right | |
| 18:54:46 | sean-k-mooney | this should be the combiantion fo stephens change + artoms | |
| 18:55:07 | sean-k-mooney | so i would expect that to contain PCPUS | |
| 18:55:22 | sean-k-mooney | i have other jobs that are just stephens code running too | |
| 18:55:55 | dansmith | wouldn't we fail to boot at all if there wasn't any inventory? | |
| 18:55:59 | dansmith | meaning, fail to get to the compute at all | |
| 18:56:55 | sean-k-mooney | i think its reporting inventory of PCPU and either not traslating or doing the fallback | |
| 18:57:06 | artom | I didn't touch any of the scheduler stuff in my code - I assumed it handled placement allocations correctly for a live migration | |
| 18:57:25 | artom | Maybe that part needs to be filled in for PCPUs? | |
| 18:58:10 | dansmith | inventory data: {'VCPU': {'total': 7, 'reserved': 0, 'min_unit': 1, 'max_unit': 7, 'step_size': 1, 'allocation_ratio': 16.0}, 'MEMORY_MB': {'total': 16039, 'reserved': 512, 'min_unit': 1, 'max_unit': 16039, 'step_size': 1, 'allocation_ratio': 1.5}, 'DISK_GB': {'total': 74, 'reserved': 0, 'min_unit': 1, 'max_unit': 74, 'step_size': 1, 'allocation_ratio': 1.0} | |
| 18:58:14 | dansmith | am I missing the PCPU in there? | |
| 18:59:18 | dansmith | Translating request for VCPU=2 to VCPU=0,PCPU=2 | |
| 18:59:44 | sean-k-mooney | no | |
| 18:59:44 | dansmith | I thought that was supposed to happen in the scheduler request, but that's in the compute log | |
| 19:00:06 | sean-k-mooney | so did i | |
| 19:00:38 | sean-k-mooney | could it happen in the compute on resize maybe | |
| 19:00:46 | dansmith | no, it can't call the scheduler | |
| 19:00:59 | sean-k-mooney | well its in the scheduler utils | |
| 19:01:05 | sean-k-mooney | but ya | |
| 19:01:13 | dansmith | unless that's just being logged from a util method we're calling to generate an allocation update or something | |
| 19:01:19 | sean-k-mooney | do we have placmenet logs | |
| 19:02:43 | sean-k-mooney | that is a request for PCPUs right https://zuul.opendev.org/t/openstack/build/c1f6ccc551c649bab6f60911fd3485ff/log/controller/logs/screen-placement-api.txt.gz#342 | |
| 19:03:13 | sean-k-mooney | its using the fallback | |
| 19:03:37 | dansmith | that's a req from the scheduler I assume, | |
| 19:03:46 | dansmith | I'm not sure why we're seeing the translation message in the compute log tho | |
| 19:04:18 | sean-k-mooney | yes and just below it we see it request again with VCPUS | |
| 19:05:22 | sean-k-mooney | stephenfin: did i miss adding something in the nova conf | |
| 19:05:33 | stephenfin | sean-k-mooney: what did you add? | |
| 19:05:55 | sean-k-mooney | just vcpu_pin_set in this case + cpu_shared_set | |
| 19:06:07 | sean-k-mooney | but it hsould still get PCPU no? | |
| 19:06:14 | stephenfin | nope | |
| 19:06:36 | dansmith | stephenfin: do you know why we're logging that translation thing in the compute log? is it from generating allocations to compare in the periodic or something? | |
| 19:06:36 | sean-k-mooney | oh ok | |
| 19:06:37 | stephenfin | the host won't do reshape or report PCPU until '[compute] cpu_dedicated_set' is configured. That's our switchover | |
| 19:06:54 | sean-k-mooney | ah ok | |
| 19:07:03 | sean-k-mooney | ill update that job then | |
| 19:07:12 | stephenfin | dansmith: which one? | |
| 19:07:15 | sean-k-mooney | the other use cpu_share_set and cpu_dedicated_set | |
| 19:07:20 | stephenfin | oh Translating request for VCPU=2 to VCPU=0,PCPU=2 | |
| 19:07:48 | stephenfin | yeah, sec | |
| 19:07:48 | dansmith | yeah | |
| 19:08:14 | dansmith | I saw that, forgot it was in the compute log and was like "cool, cool, we're ... wait a sec." :) | |
| 19:09:16 | stephenfin | so this is the code that does the translation https://review.opendev.org/#/c/671801/48/nova/scheduler/utils.py | |
| 19:09:34 | stephenfin | and the question is where do we build ResourceRequest object on the compute ndoe | |
| 19:09:36 | stephenfin | *node | |
| 19:09:43 | stephenfin | because I didn't add that :) | |
| 19:09:54 | dansmith | if we're legit logging that we should probably change that wording a little | |
| 19:10:32 | stephenfin | got it | |
| 19:10:45 | stephenfin | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L595-L602 | |
| 19:10:50 | sean-k-mooney | "cpu_dedicated_set: 0-6" is the correct spelling yes | |
| 19:11:02 | stephenfin | yup, and it has to be in the compute group | |
| 19:11:12 | sean-k-mooney | ya | |
| 19:12:10 | stephenfin | dansmith: That's from the vCPU model selection code and it was added as a quick way to extract the traits | |
| 19:12:54 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] numa + pcpus in placment live migration tests https://review.opendev.org/681771 | |