| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-13 | |||
| 10:11:06 | bauzas | sean-k-mooney: yeah hence the need of a nova-neutron discussion at the PTG | |
| 10:11:35 | sean-k-mooney | https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/network/model.py#L560-L579 this is what i have tried to do lately since the last time they rejected adding the events | |
| 10:12:11 | sean-k-mooney | with has_bind_time basically being https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/network/model.py#L481-L490 | |
| 10:13:24 | sean-k-mooney | bauzas: im happy for there to be a PTG discussion on this since i planned to bring this up anyway as part fo the ovn migration disucssion | |
| 10:13:37 | bauzas | cool | |
| 10:13:39 | sean-k-mooney | bauzas: but i just dont have high confidence that anything will be done about it | |
| 10:14:07 | sean-k-mooney | its at least the 4th time we will have talked about it | |
| 10:14:53 | bauzas | it's a yet again "Dare to Care" discussion, heh | |
| 10:15:11 | sean-k-mooney | if we really want to fix this perhaps we (nova pepole) might need to go implement it in neuton | |
| 10:15:18 | bauzas | well | |
| 10:15:37 | bauzas | first, let's see what's coming | |
| 10:16:06 | sean-k-mooney | for reasons i know that our neturon folk at redhat are proposing adding plug time event support to ovn | |
| 10:16:13 | sean-k-mooney | reason being live migration | |
| 10:16:28 | sean-k-mooney | but to actully fix ovn live migration we need to actully change ovn too | |
| 10:17:01 | sean-k-mooney | the way ovn is currently desigined its not possibel to have 0 down time live migration with libvirt | |
| 10:26:10 | lpetrut | hi, could you please take another look over the gmr patch? https://review.opendev.org/c/openstack/nova/+/810922 | |
| 10:27:27 | sean-k-mooney | sure | |
| 10:27:56 | lpetrut | thanks | |
| 10:31:22 | bauzas | lpetrut: excellent catch | |
| 10:31:49 | bauzas | lpetrut: could you please fill a bug against gmr not working properly due to uswsgi | |
| 10:32:05 | bauzas | I'd like this to be documented in the yoga relnotes if we merge it | |
| 10:32:27 | sean-k-mooney | bauzas: its not really a GMR bug | |
| 10:33:05 | sean-k-mooney | you already found that we can pass signals to the python app if we configure uwsgi correctly | |
| 10:33:30 | sean-k-mooney | but by default it will trap the sig_usr2 | |
| 10:43:12 | lpetrut | I've added a Nova release note. indeed, it doesn't seem like an oslo.reports bug, if needed I can file a bug against nova. last time, the consensus was that it's a minor feature that doesn't require a blueprint | |
| 10:53:12 | opendevreview | Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922 | |
| 11:26:42 | gibi | sean-k-mooney: I made a bit of progress witht the unshelve functional test. There is a reschedule happening, but it feels like it is on an instance from another test case?! see my last comment with logs in https://review.opendev.org/c/openstack/nova/+/813674 | |
| 11:33:52 | gibi | sean-k-mooney: the subunit file has a lot more information that what is visible from the job-output.txt | |
| 11:34:31 | gibi | it can be extracted to individual tests and it shows that the instance uuid logged in our failed test is actually mentioned in another test case log as well | |
| 11:34:34 | sean-k-mooney | gibi: dod upi see https://review.opendev.org/c/openstack/nova/+/813695 | |
| 11:35:10 | sean-k-mooney | ah you did | |
| 11:35:45 | sean-k-mooney | gibi: i feel like when we are waiting for the virsion notificiaotn in that case we need to wait for one related to the vm we are unshlving | |
| 11:36:04 | sean-k-mooney | but i tought the notifier was per test instnace | |
| 11:36:18 | gibi | sean-k-mooney: the notifier should be unique per test case too | |
| 11:36:22 | gibi | https://paste.opendev.org/show/809961/ | |
| 11:36:24 | gibi | see this paste | |
| 11:36:33 | gibi | this mentions the same uuid in two test case logs | |
| 11:36:38 | sean-k-mooney | the fact that there are 4 in my case feels supiocisly because 4 tests run when i filter | |
| 11:36:53 | sean-k-mooney | i.e. if i fileter by test_unshelve_offloaded_server_with_qos_port_pci_update_fails | |
| 11:37:00 | sean-k-mooney | it runs 4 versions of that test | |
| 11:37:01 | songwenping_ | Hi,team, when i once put two nodes in one aggregate, nova sheduler only update one node to the aggregate, if i use the other node to create vm, there are no valid host failed, the log is AZFilter return 0 hosts. | |
| 11:37:11 | gibi | sean-k-mooney: yepp there are 4 versions | |
| 11:37:27 | gibi | sean-k-mooney: if you see the 4 test case interacts via the notifier that is also a problem | |
| 11:37:30 | gibi | is should not | |
| 11:37:52 | sean-k-mooney | gibi: so im not sure that they do but i could print all of the notificaiotn object i guess | |
| 11:38:07 | sean-k-mooney | see what they are | |
| 11:38:12 | gibi | yeah that could help | |
| 11:44:03 | songwenping_ | sean-k-mooney, gibi: when does nova-scheduler update host aggregate map, change host from aggregate? | |
| 11:46:16 | gibi | sean-k-mooney: I checked 3 reproduction form the gate, it is always nova.tests.functional.test_servers.ServersTestV219.test_description_errors test case logs that mentions the same instance uuid as the failed unshelve test case | |
| 11:46:47 | gibi | songwenping_: I don't know without looking into the code, sorry | |
| 11:47:03 | sean-k-mooney | gibi: interesting so to repoduce this we shoudl run both of those tests | |
| 11:47:31 | sean-k-mooney | it does sound like we are sharing global state some how | |
| 11:48:31 | gibi | sean-k-mooney: yeah, it is alway 61 seconds after the succesfull run of .ServersTestV219.test_description_errors that the unshelve test fails | |
| 11:48:51 | gibi | that sounds like a 60 sec timout on an RPC | |
| 11:49:02 | songwenping_ | thanks gibi. :( | |
| 11:49:16 | gibi | songwenping_: sorry I knee deep in someting else at the moment | |
| 11:50:02 | sean-k-mooney | songwenping_: nova does not move host between aggreates. you have to use the api to do that | |
| 11:50:03 | gibi | sean-k-mooney: so my assumption is that the test_description_errors case does not end cleanly | |
| 11:50:30 | sean-k-mooney | songwenping_: i dont think we cache that infor in the scheduler its not part of the host state objects | |
| 11:50:43 | sean-k-mooney | songwenping_: so once its commited to the db i think the schduler will see it | |
| 11:52:00 | songwenping_ | sean-k-mooney: nova-scheduler update part of these hosts. | |
| 11:54:25 | sean-k-mooney | songwenping_: im not sure what you mean by that | |
| 11:54:56 | sean-k-mooney | the nova scedular nerver modfies aggreates or compute nodes | |
| 11:55:23 | sean-k-mooney | it just makes claims in plcement and selesct the destionation for instances | |
| 11:55:53 | sean-k-mooney | aggreate membership si entrily mange extrenally by the nova-api | |
| 11:55:56 | songwenping_ | sean-k-mooney: nova-scheduler update its host_aggregate_map attribute | |
| 11:56:23 | songwenping_ | nova-api change the aggregate's host | |
| 11:56:55 | songwenping_ | let me debug on my env first. | |
| 11:58:11 | sean-k-mooney | what release of openstack are you using | |
| 11:58:25 | songwenping_ | R | |
| 11:58:30 | sean-k-mooney | rocky | |
| 11:58:33 | songwenping_ | yes | |
| 11:58:48 | sean-k-mooney | im not sure that exist in the host manager anymore | |
| 11:59:43 | sean-k-mooney | ok it does | |
| 11:59:45 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L353 | |
| 12:00:15 | sean-k-mooney | its plural host_aggregates_map not host_aggregate_map | |
| 12:00:43 | songwenping_ | yes sorry | |
| 12:01:14 | sean-k-mooney | so its called here https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/scheduler/manager.py#L662-L670 | |
| 12:02:13 | sean-k-mooney | which is called in a number of places in tghe compute api https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/compute/api.py#L6231 | |
| 12:02:51 | sean-k-mooney | this is where we update it when you add a host https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/compute/api.py#L6414 | |
| 12:04:05 | sean-k-mooney | songwenping_: so we update it after we update it in the db but before we update it in placment | |
| 12:04:44 | sean-k-mooney | songwenping_: so if you are using placment for aggres there is a very short period of time where the host is a member of an aggreate but placment doe snot know yet | |
| 12:04:56 | sean-k-mooney | however all of this happens before the api request returns | |
| 12:05:42 | songwenping_ | yeah, but the host param seems a single once | |
| 12:06:18 | sean-k-mooney | the invocation of the schduler metohd is a cast | |
| 12:06:33 | sean-k-mooney | so it might take a while after we retrun for that to propageate and the schdulers to update | |
| 12:10:59 | songwenping_ | sean-k-moony: not exactly, we wait a long time. | |
| 12:12:18 | sean-k-mooney | songwenping_: what is the actul problem you are seeing | |
| 12:12:34 | sean-k-mooney | have you filed a bug describing it with logs | |
| 12:12:50 | songwenping_ | not yet, i am not sure | |
| 12:12:51 | sean-k-mooney | i currently on a call so i dont really have time to help debug it now | |
| 12:13:09 | songwenping_ | no hurry. | |
| 12:14:55 | songwenping_ | i am debugging now. | |
| 12:48:46 | gibi | sean-k-mooney: seems test_description_errors can fail and leave running greenthreads behind: https://paste.opendev.org/show/809966/ | |
| 12:49:00 | gibi | I mean can produce an error without the testcase failing | |
| 12:49:09 | gibi | then leaking running greanthreads | |
| 12:49:10 | sean-k-mooney | i see | |
| 12:49:26 | sean-k-mooney | ok and i guess that can somehow impact other tests | |
| 12:50:02 | gibi | yeah it is still strange how these tests interact. there is a global state somewhere | |
| 12:50:17 | sean-k-mooney | could there somehow be sharing betwen the fake_impl for oslo.messaging | |