| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-13 | |||
| 11:36:24 | gibi | see this paste | |
| 11:36:33 | gibi | this mentions the same uuid in two test case logs | |
| 11:36:38 | sean-k-mooney | the fact that there are 4 in my case feels supiocisly because 4 tests run when i filter | |
| 11:36:53 | sean-k-mooney | i.e. if i fileter by test_unshelve_offloaded_server_with_qos_port_pci_update_fails | |
| 11:37:00 | sean-k-mooney | it runs 4 versions of that test | |
| 11:37:01 | songwenping_ | Hi,team, when i once put two nodes in one aggregate, nova sheduler only update one node to the aggregate, if i use the other node to create vm, there are no valid host failed, the log is AZFilter return 0 hosts. | |
| 11:37:11 | gibi | sean-k-mooney: yepp there are 4 versions | |
| 11:37:27 | gibi | sean-k-mooney: if you see the 4 test case interacts via the notifier that is also a problem | |
| 11:37:30 | gibi | is should not | |
| 11:37:52 | sean-k-mooney | gibi: so im not sure that they do but i could print all of the notificaiotn object i guess | |
| 11:38:07 | sean-k-mooney | see what they are | |
| 11:38:12 | gibi | yeah that could help | |
| 11:44:03 | songwenping_ | sean-k-mooney, gibi: when does nova-scheduler update host aggregate map, change host from aggregate? | |
| 11:46:16 | gibi | sean-k-mooney: I checked 3 reproduction form the gate, it is always nova.tests.functional.test_servers.ServersTestV219.test_description_errors test case logs that mentions the same instance uuid as the failed unshelve test case | |
| 11:46:47 | gibi | songwenping_: I don't know without looking into the code, sorry | |
| 11:47:03 | sean-k-mooney | gibi: interesting so to repoduce this we shoudl run both of those tests | |
| 11:47:31 | sean-k-mooney | it does sound like we are sharing global state some how | |
| 11:48:31 | gibi | sean-k-mooney: yeah, it is alway 61 seconds after the succesfull run of .ServersTestV219.test_description_errors that the unshelve test fails | |
| 11:48:51 | gibi | that sounds like a 60 sec timout on an RPC | |
| 11:49:02 | songwenping_ | thanks gibi. :( | |
| 11:49:16 | gibi | songwenping_: sorry I knee deep in someting else at the moment | |
| 11:50:02 | sean-k-mooney | songwenping_: nova does not move host between aggreates. you have to use the api to do that | |
| 11:50:03 | gibi | sean-k-mooney: so my assumption is that the test_description_errors case does not end cleanly | |
| 11:50:30 | sean-k-mooney | songwenping_: i dont think we cache that infor in the scheduler its not part of the host state objects | |
| 11:50:43 | sean-k-mooney | songwenping_: so once its commited to the db i think the schduler will see it | |
| 11:52:00 | songwenping_ | sean-k-mooney: nova-scheduler update part of these hosts. | |
| 11:54:25 | sean-k-mooney | songwenping_: im not sure what you mean by that | |
| 11:54:56 | sean-k-mooney | the nova scedular nerver modfies aggreates or compute nodes | |
| 11:55:23 | sean-k-mooney | it just makes claims in plcement and selesct the destionation for instances | |
| 11:55:53 | sean-k-mooney | aggreate membership si entrily mange extrenally by the nova-api | |
| 11:55:56 | songwenping_ | sean-k-mooney: nova-scheduler update its host_aggregate_map attribute | |
| 11:56:23 | songwenping_ | nova-api change the aggregate's host | |
| 11:56:55 | songwenping_ | let me debug on my env first. | |
| 11:58:11 | sean-k-mooney | what release of openstack are you using | |
| 11:58:25 | songwenping_ | R | |
| 11:58:30 | sean-k-mooney | rocky | |
| 11:58:33 | songwenping_ | yes | |
| 11:58:48 | sean-k-mooney | im not sure that exist in the host manager anymore | |
| 11:59:43 | sean-k-mooney | ok it does | |
| 11:59:45 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L353 | |
| 12:00:15 | sean-k-mooney | its plural host_aggregates_map not host_aggregate_map | |
| 12:00:43 | songwenping_ | yes sorry | |
| 12:01:14 | sean-k-mooney | so its called here https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/scheduler/manager.py#L662-L670 | |
| 12:02:13 | sean-k-mooney | which is called in a number of places in tghe compute api https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/compute/api.py#L6231 | |
| 12:02:51 | sean-k-mooney | this is where we update it when you add a host https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/compute/api.py#L6414 | |
| 12:04:05 | sean-k-mooney | songwenping_: so we update it after we update it in the db but before we update it in placment | |
| 12:04:44 | sean-k-mooney | songwenping_: so if you are using placment for aggres there is a very short period of time where the host is a member of an aggreate but placment doe snot know yet | |
| 12:04:56 | sean-k-mooney | however all of this happens before the api request returns | |
| 12:05:42 | songwenping_ | yeah, but the host param seems a single once | |
| 12:06:18 | sean-k-mooney | the invocation of the schduler metohd is a cast | |
| 12:06:33 | sean-k-mooney | so it might take a while after we retrun for that to propageate and the schdulers to update | |
| 12:10:59 | songwenping_ | sean-k-moony: not exactly, we wait a long time. | |
| 12:12:18 | sean-k-mooney | songwenping_: what is the actul problem you are seeing | |
| 12:12:34 | sean-k-mooney | have you filed a bug describing it with logs | |
| 12:12:50 | songwenping_ | not yet, i am not sure | |
| 12:12:51 | sean-k-mooney | i currently on a call so i dont really have time to help debug it now | |
| 12:13:09 | songwenping_ | no hurry. | |
| 12:14:55 | songwenping_ | i am debugging now. | |
| 12:48:46 | gibi | sean-k-mooney: seems test_description_errors can fail and leave running greenthreads behind: https://paste.opendev.org/show/809966/ | |
| 12:49:00 | gibi | I mean can produce an error without the testcase failing | |
| 12:49:09 | gibi | then leaking running greanthreads | |
| 12:49:10 | sean-k-mooney | i see | |
| 12:49:26 | sean-k-mooney | ok and i guess that can somehow impact other tests | |
| 12:50:02 | gibi | yeah it is still strange how these tests interact. there is a global state somewhere | |
| 12:50:17 | sean-k-mooney | could there somehow be sharing betwen the fake_impl for oslo.messaging | |
| 12:51:28 | sean-k-mooney | or the db fixture. it seams like somehost when those finally time out it trigres the db to be cleaned up | |
| 12:53:09 | sean-k-mooney | or i guess the notifcation fixture but really im just speculating | |
| 12:53:35 | gibi | I dig forward.. | |
| 12:53:39 | sean-k-mooney | at least we have something to investigate | |
| 12:54:50 | gibi | yeah | |
| 13:03:54 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 13:04:22 | opendevreview | Federico Ressi proposed openstack/nova master: Check Nova project changes with Tobiko scenario test cases https://review.opendev.org/c/openstack/nova/+/806853 | |
| 13:18:50 | kashyap | sean-k-mooney: I haven't checked the specs and the Git logs yet - do you recall what's the status of vDPA in upstream Nova? | |
| 13:19:33 | sean-k-mooney | we have partial support | |
| 13:19:42 | sean-k-mooney | we do not support any move operations | |
| 13:19:50 | sean-k-mooney | and we only support it for netwok devices | |
| 13:19:53 | kashyap | Ah, right - no move support | |
| 13:20:02 | sean-k-mooney | with ovs hardware offload | |
| 13:21:03 | sean-k-mooney | e.g. we do not support vdpa in legacy mode only in switchdev mode and the sriovnic_agent only support legacy mode | |
| 13:21:33 | sean-k-mooney | so you cant use vdpa with openstack without ovs as the contol plane using tcflower to instll flow into the nic | |
| 13:21:56 | sean-k-mooney | kashyap: was there a resaond you wanted to know | |
| 13:22:24 | sean-k-mooney | for our product it will be tech previwe in 17 until we can actully support at least some of the move ops | |
| 13:22:38 | kashyap | sean-k-mooney: Not for product - I was curious from an upstream PoV | |
| 13:23:03 | sean-k-mooney | i may add more support this cycle but hardware has been a limiting factor to working on this | |
| 13:23:04 | kashyap | The reason was just I was reading about vDPA and it reminded me of it | |
| 13:23:25 | sean-k-mooney | ack | |
| 14:23:33 | sean-k-mooney | does this look familar to anyone | |
| 14:23:35 | sean-k-mooney | https://paste.opendev.org/show/809977/ | |
| 14:23:46 | sean-k-mooney | ] GET /placement/resource_providers?in_tree=4838f356-678a-477c-a2b8-7a743ad0842c => generated 162 bytes in 5 msecs (HTTP/1.1 404) | |
| 14:24:11 | sean-k-mooney | but in the debug logs the path is wrong | |
| 14:24:12 | sean-k-mooney | 92.168.1.10 "GET /placemen//resource_providers?in_tree=4838f356-678a-477c-a2b8-7a743ad0842c" status: 404 | |
| 14:25:39 | sean-k-mooney | the apach2 config just has ProxyPass "/placement" "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api/" retry=0 | |
| 14:26:41 | sean-k-mooney | this is causeing "/usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check" to fail which breaks devstack | |
| 14:26:57 | alecorps | hello | |
| 14:27:44 | alecorps | is someone can have a look to the PR 808791 ? | |
| 14:28:14 | sean-k-mooney | PR as in pull request | |
| 14:28:21 | gibi | sean-k-mooney: you have wrong config in you apache proxy | |
| 14:28:24 | gibi | sean-k-mooney: in devstack | |
| 14:28:32 | gibi | sean-k-mooney: searching for the commit... | |
| 14:28:49 | gibi | sean-k-mooney: https://review.opendev.org/c/openstack/devstack/+/811303 | |