| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-04 | |||
| 19:21:34 | alexe9191 | I am not sure I understand why? | |
| 19:21:41 | alexe9191 | https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/filters/retry_filter.py#L34 | |
| 19:22:16 | sean-k-mooney | well spec_obj.retry will not be set | |
| 19:22:29 | sean-k-mooney | so it will return true on line 36 | |
| 19:22:57 | alexe9191 | I am not quite sure how am I ending up with this number then: | |
| 19:22:57 | alexe9191 | 2021-02-04 19:16:58.981 112479 DEBUG nova.filters [req-7daf5214-19ca-48bc-9240-7db0be15c304 03169685e1924a6fa4eee2da46335331 a0b22ddc828140beaf13bc3daaba4a93 - default default] Starting with 148 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70 | |
| 19:22:58 | alexe9191 | 2021-02-04 19:16:58.982 112479 DEBUG nova.filters [req-7daf5214-19ca-48bc-9240-7db0be15c304 03169685e1924a6fa4eee2da46335331 a0b22ddc828140beaf13bc3daaba4a93 - default default] Filter RetryFilter returned 148 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 | |
| 19:23:12 | alexe9191 | For that specefic flavor, though other flavors are returning different numbers | |
| 19:23:34 | sean-k-mooney | well in that case the RetryFilter return 148 hosts | |
| 19:23:45 | sean-k-mooney | and it was given 148 hosts | |
| 19:23:50 | sean-k-mooney | so they all were pased | |
| 19:24:02 | alexe9191 | where was it given 148 hosts from ? | |
| 19:24:14 | sean-k-mooney | from placment | |
| 19:24:28 | alexe9191 | does placement register the failed hosts? | |
| 19:24:29 | sean-k-mooney | you started with 148 hosts form placment | |
| 19:24:33 | alexe9191 | ow | |
| 19:24:34 | sean-k-mooney | no | |
| 19:25:06 | sean-k-mooney | placment basically says ( based on our request) here are the set of host that could fit the vm | |
| 19:25:13 | sean-k-mooney | then the filter refine that | |
| 19:25:32 | sean-k-mooney | placemtn in rocky is basically only lookign at ram disk and cpus | |
| 19:26:06 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: libvirt: start tracking NUMACell.socket for hosts https://review.opendev.org/c/openstack/nova/+/766816 | |
| 19:26:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: extra specs/image pros: add `socket` PCI NUMA affinity https://review.opendev.org/c/openstack/nova/+/772748 | |
| 19:26:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Add `socket` PCI NUMA affinity policy request prefilter https://review.opendev.org/c/openstack/nova/+/772749 | |
| 19:26:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: pci: implement the `socket` NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779 | |
| 19:26:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Track host NUMA topology in PCI manager https://review.opendev.org/c/openstack/nova/+/774149 | |
| 19:26:11 | alexe9191 | not the availability zone or such then ? | |
| 19:26:13 | sean-k-mooney | it does a bit more but basically of your 800 comptue nodes it said here are the 148 that could fit your vm | |
| 19:26:28 | sean-k-mooney | not in rocky by deault at least | |
| 19:26:37 | sean-k-mooney | we did add the az later | |
| 19:26:53 | alexe9191 | ok... but the interesting thing then is, when i schedule the virtual machine directly on the host it works just fine | |
| 19:27:03 | alexe9191 | so from resources point of view there are plenty | |
| 19:27:05 | sean-k-mooney | actully it can do the az in rocky https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/request_filter.py#L63 | |
| 19:27:59 | sean-k-mooney | but i think that is off by default and you use the az filter | |
| 19:28:12 | alexe9191 | I am using that | |
| 19:28:31 | alexe9191 | but I end up with hosts that are not usable for that specefic az | |
| 19:28:46 | alexe9191 | they are all from zone 2,3,4 for instance and the one i want is zone1 | |
| 19:29:04 | alexe9191 | BUt this is happening only on this flavor. | |
| 19:29:18 | sean-k-mooney | the flavor wont change teh az interaction | |
| 19:29:23 | alexe9191 | indeed | |
| 19:29:53 | alexe9191 | if I drop the az I get more hosts to start with on the retry filter though. | |
| 19:30:10 | sean-k-mooney | yes so that is placment limiting the hosts | |
| 19:30:18 | sean-k-mooney | to only those in the requeted az | |
| 19:30:41 | sean-k-mooney | its likely that one of the later fiters is failing | |
| 19:31:00 | sean-k-mooney | can you paste the fule filter logs for the spwan to http://paste.openstack.org/ | |
| 19:31:32 | alexe9191 | actually what I said was just wrong.. i end up with the same number of hosts 148 if I drop the az, the scheduling happen though cause az filter is not filtering anything out | |
| 19:31:41 | alexe9191 | yes one moment let me sanitise it | |
| 19:32:13 | sean-k-mooney | ya if you have enable the plamcent version you can also disable the az filter | |
| 19:32:31 | alexe9191 | indeed that will also be done since it can be used | |
| 19:32:36 | alexe9191 | so here is the version that works: | |
| 19:32:36 | alexe9191 | http://paste.openstack.org/show/802341/ | |
| 19:32:37 | sean-k-mooney | over the release we have slowly been moving things too placment where it makes sense | |
| 19:33:15 | sean-k-mooney | yep so you went form 148 down to 104 | |
| 19:33:25 | sean-k-mooney | then those would get weighed | |
| 19:33:58 | alexe9191 | http://paste.openstack.org/show/802342/ this is the one that does not (COmpute is reporting 0 cause those are disabled) | |
| 19:34:20 | alexe9191 | Filter AvailabilityZoneFilter returned 8 because 8 are only in zone1 | |
| 19:34:28 | alexe9191 | so I am starting with less than I should | |
| 19:34:57 | sean-k-mooney | so the first following filters RetryFilter AvailabilityZoneFilter AggregateDiskFilter, AggregateCoreFilter and AggregateRamFilter can be removed | |
| 19:35:07 | sean-k-mooney | and NUMATopologyFilter should come last | |
| 19:36:07 | sean-k-mooney | so the one that faild had most of the host elimiated by the AvailabilityZoneFilter | |
| 19:36:29 | alexe9191 | indeed, because none of the hosts in zone1 reported it self to placement to be a good match for that flavor | |
| 19:36:40 | alexe9191 | though there are plenty of space on those hosts to cover the needed resources | |
| 19:36:47 | alexe9191 | and I have no max placement in the config | |
| 19:37:11 | sean-k-mooney | it sound like you have stale allocation in plamcent then | |
| 19:37:36 | sean-k-mooney | although what i dont understand is why the az filter removed any hosts | |
| 19:37:43 | alexe9191 | anyway to make sure that this is the case? | |
| 19:37:49 | sean-k-mooney | you said you enabeld the plamcent az filtering | |
| 19:38:03 | alexe9191 | no I meant on the api request | |
| 19:38:09 | sean-k-mooney | oh ok | |
| 19:38:21 | alexe9191 | apologies for the confusion:) | |
| 19:38:29 | sean-k-mooney | no worries | |
| 19:38:38 | sean-k-mooney | am we have a heal allcoation command | |
| 19:38:45 | sean-k-mooney | dansmith: do you know if that is in rocky | |
| 19:39:02 | dansmith | not off hand | |
| 19:40:12 | sean-k-mooney | so we have https://github.com/openstack/nova/blob/7b5ac717bd338be32414ae25f60a4bfe4c94c0f4/nova/cmd/manage.py#L2121 | |
| 19:40:32 | sean-k-mooney | ya that is on rocky | |
| 19:41:14 | sean-k-mooney | so you can do nova-manage --heal-allocations i think but before you do that | |
| 19:41:33 | sean-k-mooney | alexe9191: you have the aggreate ram,disk,core filters enabled | |
| 19:41:42 | alexe9191 | indeed | |
| 19:41:45 | sean-k-mooney | alexe9191: do you manage allocation ratios by aggreate | |
| 19:42:05 | alexe9191 | more or less yes | |
| 19:42:32 | sean-k-mooney | ok that is proably the issue | |
| 19:42:41 | sean-k-mooney | http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html | |
| 19:43:00 | sean-k-mooney | we deprecated those in ocata because once you use placment you can nolonger do that | |
| 19:43:16 | sean-k-mooney | alexe9191: you have to set the allocation ratios per host | |
| 19:43:35 | alexe9191 | let me check the code I think we have that in place now | |
| 19:44:53 | sean-k-mooney | ok here are the docs on that topic if you have not got them set on each compute host | |
| 19:44:56 | sean-k-mooney | https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios | |
| 19:45:19 | alexe9191 | just checked right now and we have that on rocky | |
| 19:45:31 | alexe9191 | older versions are a different story but we are getting rid of those | |
| 19:45:35 | sean-k-mooney | ok so its in the nova.conf on each of your compute nodees | |
| 19:45:40 | alexe9191 | yes | |
| 19:46:26 | alexe9191 | docker exec -it nova_compute grep cpu /etc/nova/nova.conf | |
| 19:46:26 | alexe9191 | cpu_allocation_ratio=1.0 | |
| 19:46:48 | sean-k-mooney | ok then what you basically need to do is compare the avaiable resouces in plamcnet to those reported in the hyperviors api | |
| 19:47:21 | sean-k-mooney | if there is a missmatch due to stales allcoation the usage wont be the same in nova view and placments | |
| 19:47:50 | alexe9191 | I am actually building now a json file with the resources using openstack hypervisor show | |
| 19:47:59 | sean-k-mooney | which would be why plamcent would have elminated the host before it got to the az filter | |
| 19:49:05 | sean-k-mooney | unfortunetly im more or less done for the day so i wont be able to help you contiue debuging but my best guess is placemetn and nova are out of sync | |
| 19:49:37 | sean-k-mooney | so placment is filtering out the host in that az first. before it gets to the schduler | |
| 19:50:00 | alexe9191 | You've already sat me on the good path:) I am going to check this and check the heal command if that is the case | |