| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-04 | |||
| 18:57:16 | dansmith | I know, but I think we agreed to wait until the ceph and zuulv3 thing was resolved | |
| 18:57:25 | dansmith | I already proposed it with -W to wait on that | |
| 18:58:44 | sean-k-mooney | well mor i ment we can drop integrated-gate-compute template then contol which grenddade jobs run rom the check and gate pipelines | |
| 18:58:51 | sean-k-mooney | https://github.com/openstack/nova/blob/master/.zuul.yaml#L421 | |
| 18:59:04 | sean-k-mooney | which means we coudl jsut run - nova-grenade-multinode | |
| 18:59:21 | dansmith | https://review.opendev.org/c/openstack/tempest/+/771499 | |
| 18:59:43 | dansmith | we agreed we would wait to do that until the ceph multinode zuulv3 thing was resolved | |
| 19:00:15 | sean-k-mooney | oh i know i just was pointing out we could do it via nova if we had resovled it and not need a tempest patch | |
| 19:01:02 | dansmith | and when we discussed, gmann wanted it changed there ^, but yes, the mechanics aren't hard, it's the agreement required, and in this case, blocking on the zuulv3 conversion | |
| 19:01:26 | alexe9191 | good day everyone:) | |
| 19:01:34 | sean-k-mooney | alexe9191: o/ | |
| 19:01:35 | alexe9191 | I have a question about the retry filter in nova | |
| 19:01:50 | alexe9191 | I am wondering where does it get it's spec_obj from ? specefically this piece of code here: | |
| 19:01:50 | alexe9191 | def host_passes(self, host_state, spec_obj): | |
| 19:01:51 | alexe9191 | """Skip nodes that have already been attempted.""" | |
| 19:01:51 | alexe9191 | retry = spec_obj.retry | |
| 19:01:52 | sean-k-mooney | alexe9191: it has not been required for quite some time | |
| 19:02:24 | sean-k-mooney | alexe9191: its passed in by the filter schduler | |
| 19:02:33 | alexe9191 | but where is it stored? memory or db? | |
| 19:02:58 | sean-k-mooney | its the request spec | |
| 19:03:04 | sean-k-mooney | its builts in the api | |
| 19:03:12 | sean-k-mooney | then passed to the conductor and scudler | |
| 19:03:22 | sean-k-mooney | i belive we might have it in the api db | |
| 19:03:35 | alexe9191 | Interesting, let me check | |
| 19:03:43 | dansmith | yes, api_db | |
| 19:04:30 | alexe9191 | ok, so if a host fail, it will register it's state here in that table. | |
| 19:04:32 | alexe9191 | request_specs | |
| 19:04:45 | sean-k-mooney | no | |
| 19:04:57 | sean-k-mooney | we dont commit that to the db | |
| 19:05:21 | sean-k-mooney | we only track that in memroy i belive | |
| 19:05:31 | alexe9191 | that table is quite loaded though. | |
| 19:05:47 | sean-k-mooney | the request_sepc is used for other things | |
| 19:05:53 | sean-k-mooney | so it is saved in the db | |
| 19:06:11 | sean-k-mooney | but wee dont save the failed host in the request spec during schdule and commit that back | |
| 19:06:30 | alexe9191 | so that part right here: | |
| 19:06:31 | alexe9191 | return self.filter_handler.get_filtered_objects(self.enabled_filters, | |
| 19:06:31 | alexe9191 | hosts, spec_obj, index) | |
| 19:07:02 | alexe9191 | i see it's passing hosts but the retry filter is kicking out all of the hosts in the aggregate I am trying to schedule in. They are all healthy and they are hosting VMs, they probably had issues at some time. | |
| 19:07:21 | alexe9191 | The interesting thing, this happens only with one flavor, other flavors are returning a different results for the retryfilter. | |
| 19:08:23 | alexe9191 | I restarted nova-compute on all of those hosts but that did not change the result of the scheduling:) so I am wondering to be honest where does it saves the host state. | |
| 19:08:29 | sean-k-mooney | alexe9191: what releast of nova are you using by the way | |
| 19:08:32 | alexe9191 | rocky:) | |
| 19:08:39 | sean-k-mooney | so you have placment | |
| 19:08:59 | alexe9191 | yes | |
| 19:09:10 | sean-k-mooney | then you can disable the retry filter entirly | |
| 19:09:19 | sean-k-mooney | i belive rocky is the release we stopped using it | |
| 19:09:25 | sean-k-mooney | thats what im checking now | |
| 19:10:12 | sean-k-mooney | ah it was queens https://github.com/openstack/nova/blob/master/releasenotes/notes/deprecate-retry-filter-4d1dba39a2c21836.yaml | |
| 19:10:34 | sean-k-mooney | as part of https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/return-alternate-hosts.html | |
| 19:10:52 | sean-k-mooney | alexe9191: so on rocky you can and should disable the retry filter | |
| 19:11:22 | alexe9191 | Ok! that's good to know | |
| 19:12:03 | alexe9191 | Is there a way to mitigate the effect of the retry filter right now? restarting the nova scheduler right now is probably something that's gonna cause a lot of grief | |
| 19:12:12 | alexe9191 | We have about 800~ hosts | |
| 19:12:20 | alexe9191 | 9 schedulers | |
| 19:12:33 | sean-k-mooney | really why so manny? | |
| 19:12:49 | alexe9191 | we're thinking about cells but this is in the future plans | |
| 19:13:01 | alexe9191 | it's a big infrastructure | |
| 19:13:24 | alexe9191 | that's why I was wondering if I can empty that spec_obj from a cache/db table | |
| 19:13:34 | sean-k-mooney | still scduling is typeiclaly not the largets part of a but | |
| 19:13:43 | sean-k-mooney | infacti its typeiclly quite a small amount | |
| 19:13:53 | sean-k-mooney | 9 schdulers is quite a lot | |
| 19:14:27 | sean-k-mooney | alexe9191: no the previsouly tired host are only updated in memory | |
| 19:14:51 | alexe9191 | in the nova-scheduler I am guessing | |
| 19:15:02 | sean-k-mooney | yep | |
| 19:15:25 | openstackgerrit | Merged openstack/nova master: Fix invalid argument formatting in exception messages https://review.opendev.org/c/openstack/nova/+/763511 | |
| 19:15:27 | sean-k-mooney | they only are tracked during the singel scheduling request | |
| 19:15:29 | alexe9191 | one more question, is that spec_obj also tied to the flavor ? | |
| 19:15:53 | sean-k-mooney | kind of | |
| 19:16:00 | dansmith | sean-k-mooney: should the retry filter even be used anymore? don't we pass selected_hosts down and have the conductor just iterate over them and then declare it dead? | |
| 19:16:35 | sean-k-mooney | dansmith: we do from queens | |
| 19:16:42 | sean-k-mooney | dansmith: thats whyi said to remove it | |
| 19:16:48 | dansmith | oh I see you said that above | |
| 19:16:57 | sean-k-mooney | dansmith: also i delete it on master a few release ago | |
| 19:17:14 | sean-k-mooney | alexe9191: this is the request_spec https://github.com/openstack/nova/blob/stable/rocky/nova/objects/request_spec.py#L50-L87 | |
| 19:17:34 | sean-k-mooney | it has the flavor and image avaiabel which the filters can use | |
| 19:17:35 | alexe9191 | :) So the flavor is a part of it | |
| 19:17:46 | alexe9191 | a combination of things then | |
| 19:18:03 | sean-k-mooney | its modeling all the requirements for scduling an instnace more or less | |
| 19:18:47 | alexe9191 | that's the reason why it's giving different results for different flavors | |
| 19:18:59 | sean-k-mooney | yes | |
| 19:19:16 | alexe9191 | :) Many thanks! I am gonna go and schedule a remove of the retry filter since it's not needed | |
| 19:19:17 | sean-k-mooney | although on the first iteration thorugh the scudler it should do nothing | |
| 19:19:48 | alexe9191 | it's doing that on the retryfilter | |
| 19:20:11 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/filters/retry_filter.py#L43 | |
| 19:20:22 | sean-k-mooney | retry.hosts shoudl be empty | |
| 19:21:34 | alexe9191 | I am not sure I understand why? | |
| 19:21:41 | alexe9191 | https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/filters/retry_filter.py#L34 | |
| 19:22:16 | sean-k-mooney | well spec_obj.retry will not be set | |
| 19:22:29 | sean-k-mooney | so it will return true on line 36 | |
| 19:22:57 | alexe9191 | I am not quite sure how am I ending up with this number then: | |
| 19:22:57 | alexe9191 | 2021-02-04 19:16:58.981 112479 DEBUG nova.filters [req-7daf5214-19ca-48bc-9240-7db0be15c304 03169685e1924a6fa4eee2da46335331 a0b22ddc828140beaf13bc3daaba4a93 - default default] Starting with 148 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:70 | |
| 19:22:58 | alexe9191 | 2021-02-04 19:16:58.982 112479 DEBUG nova.filters [req-7daf5214-19ca-48bc-9240-7db0be15c304 03169685e1924a6fa4eee2da46335331 a0b22ddc828140beaf13bc3daaba4a93 - default default] Filter RetryFilter returned 148 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104 | |
| 19:23:12 | alexe9191 | For that specefic flavor, though other flavors are returning different numbers | |
| 19:23:34 | sean-k-mooney | well in that case the RetryFilter return 148 hosts | |
| 19:23:45 | sean-k-mooney | and it was given 148 hosts | |
| 19:23:50 | sean-k-mooney | so they all were pased | |
| 19:24:02 | alexe9191 | where was it given 148 hosts from ? | |
| 19:24:14 | sean-k-mooney | from placment | |
| 19:24:28 | alexe9191 | does placement register the failed hosts? | |
| 19:24:29 | sean-k-mooney | you started with 148 hosts form placment | |