Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-13
09:51:23 sean-k-mooney bauzas: its more a contract
09:52:16 sean-k-mooney bauzas: i proposed that neutron tell use when they send events or normalise to a common contract back in train and neutron did not agree to either
09:52:49 opendevreview Alexey Stupnikov proposed openstack/nova master: Rollback problematic port bindings on source host only https://review.opendev.org/c/openstack/nova/+/813659
09:55:34 sean-k-mooney bauzas: you can see in the first version fo thsi we had a network_events section https://review.opendev.org/c/openstack/neutron-specs/+/645173/1/specs/train/port-binding-extended-information.rst#141
09:58:40 sean-k-mooney bauzas: this is the orginal etherpad i wrote with rodolfo https://etherpad.opendev.org/p/portbinding-records
09:59:38 sean-k-mooney it was coverd durign the train corss project session https://etherpad.opendev.org/p/ptg-train-xproj-nova-neutron
10:00:47 sean-k-mooney bauzas: i am fine with finding a better solution to what i proposed in thte past however nova must know the behaivor of the neutron backend to operat correctly
10:01:39 sean-k-mooney what we are doing today is complex and very error prone and i spend far too much of my time currently fixing bugs that are cause by not knowing when events will be sent
10:06:49 bauzas sean-k-mooney: that's my concern
10:06:59 bauzas the more nova needs to know about neutron, the more issues we could get
10:07:08 bauzas as operators need to set different options per service
10:07:21 bauzas and they can miss some related options
10:07:53 sean-k-mooney bauzas: i have been trying to get neuton to expose the info we need for litrally year at this point with the goal of ensureing that no config option are need in nova
10:08:17 sean-k-mooney this has been an ongoing battle since before placment or os-vif was a thing
10:08:27 sean-k-mooney so i agree
10:08:37 sean-k-mooney we shoudl not need operators to care or set anything
10:09:09 sean-k-mooney but to do that we must know either what contract the neutron backend has abstractly or we need to embed that knolage in nova and just know what backend it is
10:09:35 sean-k-mooney currently we try tro guess based on the vif_type and some other partmeter in the port binding_details field
10:09:49 sean-k-mooney but we dont really have the info we need
10:11:06 bauzas sean-k-mooney: yeah hence the need of a nova-neutron discussion at the PTG
10:11:35 sean-k-mooney https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/network/model.py#L560-L579 this is what i have tried to do lately since the last time they rejected adding the events
10:12:11 sean-k-mooney with has_bind_time basically being https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/network/model.py#L481-L490
10:13:24 sean-k-mooney bauzas: im happy for there to be a PTG discussion on this since i planned to bring this up anyway as part fo the ovn migration disucssion
10:13:37 bauzas cool
10:13:39 sean-k-mooney bauzas: but i just dont have high confidence that anything will be done about it
10:14:07 sean-k-mooney its at least the 4th time we will have talked about it
10:14:53 bauzas it's a yet again "Dare to Care" discussion, heh
10:15:11 sean-k-mooney if we really want to fix this perhaps we (nova pepole) might need to go implement it in neuton
10:15:18 bauzas well
10:15:37 bauzas first, let's see what's coming
10:16:06 sean-k-mooney for reasons i know that our neturon folk at redhat are proposing adding plug time event support to ovn
10:16:13 sean-k-mooney reason being live migration
10:16:28 sean-k-mooney but to actully fix ovn live migration we need to actully change ovn too
10:17:01 sean-k-mooney the way ovn is currently desigined its not possibel to have 0 down time live migration with libvirt
10:26:10 lpetrut hi, could you please take another look over the gmr patch? https://review.opendev.org/c/openstack/nova/+/810922
10:27:27 sean-k-mooney sure
10:27:56 lpetrut thanks
10:31:22 bauzas lpetrut: excellent catch
10:31:49 bauzas lpetrut: could you please fill a bug against gmr not working properly due to uswsgi
10:32:05 bauzas I'd like this to be documented in the yoga relnotes if we merge it
10:32:27 sean-k-mooney bauzas: its not really a GMR bug
10:33:05 sean-k-mooney you already found that we can pass signals to the python app if we configure uwsgi correctly
10:33:30 sean-k-mooney but by default it will trap the sig_usr2
10:43:12 lpetrut I've added a Nova release note. indeed, it doesn't seem like an oslo.reports bug, if needed I can file a bug against nova. last time, the consensus was that it's a minor feature that doesn't require a blueprint
10:53:12 opendevreview Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922
11:26:42 gibi sean-k-mooney: I made a bit of progress witht the unshelve functional test. There is a reschedule happening, but it feels like it is on an instance from another test case?! see my last comment with logs in https://review.opendev.org/c/openstack/nova/+/813674
11:33:52 gibi sean-k-mooney: the subunit file has a lot more information that what is visible from the job-output.txt
11:34:31 gibi it can be extracted to individual tests and it shows that the instance uuid logged in our failed test is actually mentioned in another test case log as well
11:34:34 sean-k-mooney gibi: dod upi see https://review.opendev.org/c/openstack/nova/+/813695
11:35:10 sean-k-mooney ah you did
11:35:45 sean-k-mooney gibi: i feel like when we are waiting for the virsion notificiaotn in that case we need to wait for one related to the vm we are unshlving
11:36:04 sean-k-mooney but i tought the notifier was per test instnace
11:36:18 gibi sean-k-mooney: the notifier should be unique per test case too
11:36:22 gibi https://paste.opendev.org/show/809961/
11:36:24 gibi see this paste
11:36:33 gibi this mentions the same uuid in two test case logs
11:36:38 sean-k-mooney the fact that there are 4 in my case feels supiocisly because 4 tests run when i filter
11:36:53 sean-k-mooney i.e. if i fileter by test_unshelve_offloaded_server_with_qos_port_pci_update_fails
11:37:00 sean-k-mooney it runs 4 versions of that test
11:37:01 songwenping_ Hi,team, when i once put two nodes in one aggregate, nova sheduler only update one node to the aggregate, if i use the other node to create vm, there are no valid host failed, the log is AZFilter return 0 hosts.
11:37:11 gibi sean-k-mooney: yepp there are 4 versions
11:37:27 gibi sean-k-mooney: if you see the 4 test case interacts via the notifier that is also a problem
11:37:30 gibi is should not
11:37:52 sean-k-mooney gibi: so im not sure that they do but i could print all of the notificaiotn object i guess
11:38:07 sean-k-mooney see what they are
11:38:12 gibi yeah that could help
11:44:03 songwenping_ sean-k-mooney, gibi: when does nova-scheduler update host aggregate map, change host from aggregate?
11:46:16 gibi sean-k-mooney: I checked 3 reproduction form the gate, it is always nova.tests.functional.test_servers.ServersTestV219.test_description_errors test case logs that mentions the same instance uuid as the failed unshelve test case
11:46:47 gibi songwenping_: I don't know without looking into the code, sorry
11:47:03 sean-k-mooney gibi: interesting so to repoduce this we shoudl run both of those tests
11:47:31 sean-k-mooney it does sound like we are sharing global state some how
11:48:31 gibi sean-k-mooney: yeah, it is alway 61 seconds after the succesfull run of .ServersTestV219.test_description_errors that the unshelve test fails
11:48:51 gibi that sounds like a 60 sec timout on an RPC
11:49:02 songwenping_ thanks gibi. :(
11:49:16 gibi songwenping_: sorry I knee deep in someting else at the moment
11:50:02 sean-k-mooney songwenping_: nova does not move host between aggreates. you have to use the api to do that
11:50:03 gibi sean-k-mooney: so my assumption is that the test_description_errors case does not end cleanly
11:50:30 sean-k-mooney songwenping_: i dont think we cache that infor in the scheduler its not part of the host state objects
11:50:43 sean-k-mooney songwenping_: so once its commited to the db i think the schduler will see it
11:52:00 songwenping_ sean-k-mooney: nova-scheduler update part of these hosts.
11:54:25 sean-k-mooney songwenping_: im not sure what you mean by that
11:54:56 sean-k-mooney the nova scedular nerver modfies aggreates or compute nodes
11:55:23 sean-k-mooney it just makes claims in plcement and selesct the destionation for instances
11:55:53 sean-k-mooney aggreate membership si entrily mange extrenally by the nova-api
11:55:56 songwenping_ sean-k-mooney: nova-scheduler update its host_aggregate_map attribute
11:56:23 songwenping_ nova-api change the aggregate's host
11:56:55 songwenping_ let me debug on my env first.
11:58:11 sean-k-mooney what release of openstack are you using
11:58:25 songwenping_ R
11:58:30 sean-k-mooney rocky
11:58:33 songwenping_ yes
11:58:48 sean-k-mooney im not sure that exist in the host manager anymore
11:59:43 sean-k-mooney ok it does
11:59:45 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L353
12:00:15 sean-k-mooney its plural host_aggregates_map not host_aggregate_map
12:00:43 songwenping_ yes sorry
12:01:14 sean-k-mooney so its called here https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/scheduler/manager.py#L662-L670
12:02:13 sean-k-mooney which is called in a number of places in tghe compute api https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/compute/api.py#L6231
12:02:51 sean-k-mooney this is where we update it when you add a host https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/compute/api.py#L6414
12:04:05 sean-k-mooney songwenping_: so we update it after we update it in the db but before we update it in placment

Earlier   Later