| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-04 | |||
| 19:07:02 | alexe9191 | i see it's passing hosts but the retry filter is kicking out all of the hosts in the aggregate I am trying to schedule in. They are all healthy and they are hosting VMs, they probably had issues at some time. | |
| 19:07:21 | alexe9191 | The interesting thing, this happens only with one flavor, other flavors are returning a different results for the retryfilter. | |
| 19:08:23 | alexe9191 | I restarted nova-compute on all of those hosts but that did not change the result of the scheduling:) so I am wondering to be honest where does it saves the host state. | |
| 19:08:29 | sean-k-mooney | alexe9191: what releast of nova are you using by the way | |
| 19:08:32 | alexe9191 | rocky:) | |
| 19:08:39 | sean-k-mooney | so you have placment | |
| 19:08:59 | alexe9191 | yes | |
| 19:09:10 | sean-k-mooney | then you can disable the retry filter entirly | |
| 19:09:19 | sean-k-mooney | i belive rocky is the release we stopped using it | |
| 19:09:25 | sean-k-mooney | thats what im checking now | |
| 19:10:12 | sean-k-mooney | ah it was queens https://github.com/openstack/nova/blob/master/releasenotes/notes/deprecate-retry-filter-4d1dba39a2c21836.yaml | |
| 19:10:34 | sean-k-mooney | as part of https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/return-alternate-hosts.html | |
| 19:10:52 | sean-k-mooney | alexe9191: so on rocky you can and should disable the retry filter | |
| 19:11:22 | alexe9191 | Ok! that's good to know | |
| 19:12:03 | alexe9191 | Is there a way to mitigate the effect of the retry filter right now? restarting the nova scheduler right now is probably something that's gonna cause a lot of grief | |
| 19:12:12 | alexe9191 | We have about 800~ hosts | |
| 19:12:20 | alexe9191 | 9 schedulers | |
| 19:12:33 | sean-k-mooney | really why so manny? | |
| 19:12:49 | alexe9191 | we're thinking about cells but this is in the future plans | |
| 19:13:01 | alexe9191 | it's a big infrastructure | |
| 19:13:24 | alexe9191 | that's why I was wondering if I can empty that spec_obj from a cache/db table | |
| 19:13:34 | sean-k-mooney | still scduling is typeiclaly not the largets part of a but | |
| 19:13:43 | sean-k-mooney | infacti its typeiclly quite a small amount | |
| 19:13:53 | sean-k-mooney | 9 schdulers is quite a lot | |
| 19:14:27 | sean-k-mooney | alexe9191: no the previsouly tired host are only updated in memory | |
| 19:14:51 | alexe9191 | in the nova-scheduler I am guessing | |
| 19:15:02 | sean-k-mooney | yep | |
| 19:15:25 | openstackgerrit | Merged openstack/nova master: Fix invalid argument formatting in exception messages https://review.opendev.org/c/openstack/nova/+/763511 | |
| 19:15:27 | sean-k-mooney | they only are tracked during the singel scheduling request | |
| 19:15:29 | alexe9191 | one more question, is that spec_obj also tied to the flavor ? | |
| 19:15:53 | sean-k-mooney | kind of | |
| 19:16:00 | dansmith | sean-k-mooney: should the retry filter even be used anymore? don't we pass selected_hosts down and have the conductor just iterate over them and then declare it dead? | |
| 19:16:35 | sean-k-mooney | dansmith: we do from queens | |
| 19:16:42 | sean-k-mooney | dansmith: thats whyi said to remove it | |
| 19:16:48 | dansmith | oh I see you said that above | |
| 19:16:57 | sean-k-mooney | dansmith: also i delete it on master a few release ago | |
| 19:17:14 | sean-k-mooney | alexe9191: this is the request_spec https://github.com/openstack/nova/blob/stable/rocky/nova/objects/request_spec.py#L50-L87 | |
| 19:17:34 | sean-k-mooney | it has the flavor and image avaiabel which the filters can use | |
| 19:17:35 | alexe9191 | :) So the flavor is a part of it | |
| 19:17:46 | alexe9191 | a combination of things then | |
| 19:18:03 | sean-k-mooney | its modeling all the requirements for scduling an instnace more or less | |
| 19:18:47 | alexe9191 | that's the reason why it's giving different results for different flavors | |
| 19:18:59 | sean-k-mooney | yes | |
| 19:19:16 | alexe9191 | :) Many thanks! I am gonna go and schedule a remove of the retry filter since it's not needed | |
| 19:19:17 | sean-k-mooney | although on the first iteration thorugh the scudler it should do nothing | |
| 19:19:48 | alexe9191 | it's doing that on the retryfilter | |
| 19:20:11 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/filters/retry_filter.py#L43 | |
| 19:20:22 | sean-k-mooney | retry.hosts shoudl be empty | |
| 19:21:34 | alexe9191 | I am not sure I understand why? | |
| 19:21:41 | alexe9191 | https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/filters/retry_filter.py#L34 | |
| 19:22:16 | sean-k-mooney | well spec_obj.retry will not be set | |
| 19:22:29 | sean-k-mooney | so it will return true on line 36 | |
| 19:22:57 | alexe9191 | 2021-02-04 19:16:58.981 112479 DEBUG nova.filters [req-7daf5214-19ca-48bc-9240-7db0be15c304 03169685e1924a6fa4eee2da46335331 a0b22ddc828140beaf13bc3daaba4a93 - default default] Starting with 148 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70 | |
| 19:22:57 | alexe9191 | I am not quite sure how am I ending up with this number then: | |
| 19:22:58 | alexe9191 | 2021-02-04 19:16:58.982 112479 DEBUG nova.filters [req-7daf5214-19ca-48bc-9240-7db0be15c304 03169685e1924a6fa4eee2da46335331 a0b22ddc828140beaf13bc3daaba4a93 - default default] Filter RetryFilter returned 148 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 | |
| 19:23:12 | alexe9191 | For that specefic flavor, though other flavors are returning different numbers | |
| 19:23:34 | sean-k-mooney | well in that case the RetryFilter return 148 hosts | |
| 19:23:45 | sean-k-mooney | and it was given 148 hosts | |
| 19:23:50 | sean-k-mooney | so they all were pased | |
| 19:24:02 | alexe9191 | where was it given 148 hosts from ? | |
| 19:24:14 | sean-k-mooney | from placment | |
| 19:24:28 | alexe9191 | does placement register the failed hosts? | |
| 19:24:29 | sean-k-mooney | you started with 148 hosts form placment | |
| 19:24:33 | alexe9191 | ow | |
| 19:24:34 | sean-k-mooney | no | |
| 19:25:06 | sean-k-mooney | placment basically says ( based on our request) here are the set of host that could fit the vm | |
| 19:25:13 | sean-k-mooney | then the filter refine that | |
| 19:25:32 | sean-k-mooney | placemtn in rocky is basically only lookign at ram disk and cpus | |
| 19:26:06 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: libvirt: start tracking NUMACell.socket for hosts https://review.opendev.org/c/openstack/nova/+/766816 | |
| 19:26:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Add `socket` PCI NUMA affinity policy request prefilter https://review.opendev.org/c/openstack/nova/+/772749 | |
| 19:26:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: extra specs/image pros: add `socket` PCI NUMA affinity https://review.opendev.org/c/openstack/nova/+/772748 | |
| 19:26:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Track host NUMA topology in PCI manager https://review.opendev.org/c/openstack/nova/+/774149 | |
| 19:26:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: pci: implement the `socket` NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779 | |
| 19:26:11 | alexe9191 | not the availability zone or such then ? | |
| 19:26:13 | sean-k-mooney | it does a bit more but basically of your 800 comptue nodes it said here are the 148 that could fit your vm | |
| 19:26:28 | sean-k-mooney | not in rocky by deault at least | |
| 19:26:37 | sean-k-mooney | we did add the az later | |
| 19:26:53 | alexe9191 | ok... but the interesting thing then is, when i schedule the virtual machine directly on the host it works just fine | |
| 19:27:03 | alexe9191 | so from resources point of view there are plenty | |
| 19:27:05 | sean-k-mooney | actully it can do the az in rocky https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/request_filter.py#L63 | |
| 19:27:59 | sean-k-mooney | but i think that is off by default and you use the az filter | |
| 19:28:12 | alexe9191 | I am using that | |
| 19:28:31 | alexe9191 | but I end up with hosts that are not usable for that specefic az | |
| 19:28:46 | alexe9191 | they are all from zone 2,3,4 for instance and the one i want is zone1 | |
| 19:29:04 | alexe9191 | BUt this is happening only on this flavor. | |
| 19:29:18 | sean-k-mooney | the flavor wont change teh az interaction | |
| 19:29:23 | alexe9191 | indeed | |
| 19:29:53 | alexe9191 | if I drop the az I get more hosts to start with on the retry filter though. | |
| 19:30:10 | sean-k-mooney | yes so that is placment limiting the hosts | |
| 19:30:18 | sean-k-mooney | to only those in the requeted az | |
| 19:30:41 | sean-k-mooney | its likely that one of the later fiters is failing | |
| 19:31:00 | sean-k-mooney | can you paste the fule filter logs for the spwan to http://paste.openstack.org/ | |
| 19:31:32 | alexe9191 | actually what I said was just wrong.. i end up with the same number of hosts 148 if I drop the az, the scheduling happen though cause az filter is not filtering anything out | |
| 19:31:41 | alexe9191 | yes one moment let me sanitise it | |
| 19:32:13 | sean-k-mooney | ya if you have enable the plamcent version you can also disable the az filter | |
| 19:32:31 | alexe9191 | indeed that will also be done since it can be used | |
| 19:32:36 | alexe9191 | http://paste.openstack.org/show/802341/ | |
| 19:32:36 | alexe9191 | so here is the version that works: | |
| 19:32:37 | sean-k-mooney | over the release we have slowly been moving things too placment where it makes sense | |
| 19:33:15 | sean-k-mooney | yep so you went form 148 down to 104 | |