Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-13
07:40:56 bauzas nova shouldn't know which backend neutron uses
07:40:59 gibi bauzas: yes that would be also an option but I think that is a harder one
07:41:00 bauzas at least for an event
07:41:41 bauzas I don't understand why it'd be hard for neutron to provide an event for plugs
07:42:02 bauzas again, having events be different between neutron backends looks bad to me
07:44:40 bauzas gibi: just added a point about this in https://etherpad.opendev.org/p/nova-yoga-ptg L105
07:49:10 gibi bauzas: thanks I will try to fill in the details htere
07:49:12 gibi sean-k-mooney: ^^
07:49:13 gibi fy
07:49:15 gibi fyi
08:42:58 frickler bauzas: lots of jobs with POST_FAILURE for failing log upload, but they seem to be back now, so safe to recheck hopefully
08:44:16 bauzas frickler: yeah the outage is now done
08:44:40 bauzas they isolated the faulty DC
08:45:19 bauzas thread in French https://twitter.com/olesovhcom/status/1448196879020433409
08:47:55 opendevreview Elod Illes proposed openstack/nova stable/ussuri: Revert "[stable-only] Set lower-constraints job as non-voting" https://review.opendev.org/c/openstack/nova/+/813784
09:45:52 sean-k-mooney bauzas: nova really really really need to know what what backend neutron uses
09:46:17 sean-k-mooney bauzas: or they need to tell use exactly when they will send the envents for that backend opacly
09:47:01 sean-k-mooney bauzas: not all backend can send plug time events because they dont all have a way to send info too neutron about when the port is created and its finsihed wiring it up
09:47:24 sean-k-mooney bauzas: at present ovn cannot do that although the ml2 dirver can be modifed to eventually do that
09:48:04 sean-k-mooney bauzas: for odl they had to create a websocket on the netorn server and modify odl to seend events to that websocket to implemente plug time events whihc invovled a lot fo code development in odl
09:48:41 sean-k-mooney bauzas: contrail still has not implemnted multiple port binding gettingthem to send plug time events likely will never happen
09:49:27 sean-k-mooney linux bridge can send pulg time event but because it currently pool for interface chcnages insted of opening a netlink socket and reciving event it can miss the removal and addtion of interface on hard reboot
09:50:40 bauzas sean-k-mooney: sorry but for me, events are like notifications
09:50:49 bauzas sean-k-mooney: it's kinda a public API
09:51:23 sean-k-mooney bauzas: its more a contract
09:52:16 sean-k-mooney bauzas: i proposed that neutron tell use when they send events or normalise to a common contract back in train and neutron did not agree to either
09:52:49 opendevreview Alexey Stupnikov proposed openstack/nova master: Rollback problematic port bindings on source host only https://review.opendev.org/c/openstack/nova/+/813659
09:55:34 sean-k-mooney bauzas: you can see in the first version fo thsi we had a network_events section https://review.opendev.org/c/openstack/neutron-specs/+/645173/1/specs/train/port-binding-extended-information.rst#141
09:58:40 sean-k-mooney bauzas: this is the orginal etherpad i wrote with rodolfo https://etherpad.opendev.org/p/portbinding-records
09:59:38 sean-k-mooney it was coverd durign the train corss project session https://etherpad.opendev.org/p/ptg-train-xproj-nova-neutron
10:00:47 sean-k-mooney bauzas: i am fine with finding a better solution to what i proposed in thte past however nova must know the behaivor of the neutron backend to operat correctly
10:01:39 sean-k-mooney what we are doing today is complex and very error prone and i spend far too much of my time currently fixing bugs that are cause by not knowing when events will be sent
10:06:49 bauzas sean-k-mooney: that's my concern
10:06:59 bauzas the more nova needs to know about neutron, the more issues we could get
10:07:08 bauzas as operators need to set different options per service
10:07:21 bauzas and they can miss some related options
10:07:53 sean-k-mooney bauzas: i have been trying to get neuton to expose the info we need for litrally year at this point with the goal of ensureing that no config option are need in nova
10:08:17 sean-k-mooney this has been an ongoing battle since before placment or os-vif was a thing
10:08:27 sean-k-mooney so i agree
10:08:37 sean-k-mooney we shoudl not need operators to care or set anything
10:09:09 sean-k-mooney but to do that we must know either what contract the neutron backend has abstractly or we need to embed that knolage in nova and just know what backend it is
10:09:35 sean-k-mooney currently we try tro guess based on the vif_type and some other partmeter in the port binding_details field
10:09:49 sean-k-mooney but we dont really have the info we need
10:11:06 bauzas sean-k-mooney: yeah hence the need of a nova-neutron discussion at the PTG
10:11:35 sean-k-mooney https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/network/model.py#L560-L579 this is what i have tried to do lately since the last time they rejected adding the events
10:12:11 sean-k-mooney with has_bind_time basically being https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/network/model.py#L481-L490
10:13:24 sean-k-mooney bauzas: im happy for there to be a PTG discussion on this since i planned to bring this up anyway as part fo the ovn migration disucssion
10:13:37 bauzas cool
10:13:39 sean-k-mooney bauzas: but i just dont have high confidence that anything will be done about it
10:14:07 sean-k-mooney its at least the 4th time we will have talked about it
10:14:53 bauzas it's a yet again "Dare to Care" discussion, heh
10:15:11 sean-k-mooney if we really want to fix this perhaps we (nova pepole) might need to go implement it in neuton
10:15:18 bauzas well
10:15:37 bauzas first, let's see what's coming
10:16:06 sean-k-mooney for reasons i know that our neturon folk at redhat are proposing adding plug time event support to ovn
10:16:13 sean-k-mooney reason being live migration
10:16:28 sean-k-mooney but to actully fix ovn live migration we need to actully change ovn too
10:17:01 sean-k-mooney the way ovn is currently desigined its not possibel to have 0 down time live migration with libvirt
10:26:10 lpetrut hi, could you please take another look over the gmr patch? https://review.opendev.org/c/openstack/nova/+/810922
10:27:27 sean-k-mooney sure
10:27:56 lpetrut thanks
10:31:22 bauzas lpetrut: excellent catch
10:31:49 bauzas lpetrut: could you please fill a bug against gmr not working properly due to uswsgi
10:32:05 bauzas I'd like this to be documented in the yoga relnotes if we merge it
10:32:27 sean-k-mooney bauzas: its not really a GMR bug
10:33:05 sean-k-mooney you already found that we can pass signals to the python app if we configure uwsgi correctly
10:33:30 sean-k-mooney but by default it will trap the sig_usr2
10:43:12 lpetrut I've added a Nova release note. indeed, it doesn't seem like an oslo.reports bug, if needed I can file a bug against nova. last time, the consensus was that it's a minor feature that doesn't require a blueprint
10:53:12 opendevreview Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922
11:26:42 gibi sean-k-mooney: I made a bit of progress witht the unshelve functional test. There is a reschedule happening, but it feels like it is on an instance from another test case?! see my last comment with logs in https://review.opendev.org/c/openstack/nova/+/813674
11:33:52 gibi sean-k-mooney: the subunit file has a lot more information that what is visible from the job-output.txt
11:34:31 gibi it can be extracted to individual tests and it shows that the instance uuid logged in our failed test is actually mentioned in another test case log as well
11:34:34 sean-k-mooney gibi: dod upi see https://review.opendev.org/c/openstack/nova/+/813695
11:35:10 sean-k-mooney ah you did
11:35:45 sean-k-mooney gibi: i feel like when we are waiting for the virsion notificiaotn in that case we need to wait for one related to the vm we are unshlving
11:36:04 sean-k-mooney but i tought the notifier was per test instnace
11:36:18 gibi sean-k-mooney: the notifier should be unique per test case too
11:36:22 gibi https://paste.opendev.org/show/809961/
11:36:24 gibi see this paste
11:36:33 gibi this mentions the same uuid in two test case logs
11:36:38 sean-k-mooney the fact that there are 4 in my case feels supiocisly because 4 tests run when i filter
11:36:53 sean-k-mooney i.e. if i fileter by test_unshelve_offloaded_server_with_qos_port_pci_update_fails
11:37:00 sean-k-mooney it runs 4 versions of that test
11:37:01 songwenping_ Hi,team, when i once put two nodes in one aggregate, nova sheduler only update one node to the aggregate, if i use the other node to create vm, there are no valid host failed, the log is AZFilter return 0 hosts.
11:37:11 gibi sean-k-mooney: yepp there are 4 versions
11:37:27 gibi sean-k-mooney: if you see the 4 test case interacts via the notifier that is also a problem
11:37:30 gibi is should not
11:37:52 sean-k-mooney gibi: so im not sure that they do but i could print all of the notificaiotn object i guess
11:38:07 sean-k-mooney see what they are
11:38:12 gibi yeah that could help
11:44:03 songwenping_ sean-k-mooney, gibi: when does nova-scheduler update host aggregate map, change host from aggregate?
11:46:16 gibi sean-k-mooney: I checked 3 reproduction form the gate, it is always nova.tests.functional.test_servers.ServersTestV219.test_description_errors test case logs that mentions the same instance uuid as the failed unshelve test case
11:46:47 gibi songwenping_: I don't know without looking into the code, sorry
11:47:03 sean-k-mooney gibi: interesting so to repoduce this we shoudl run both of those tests
11:47:31 sean-k-mooney it does sound like we are sharing global state some how
11:48:31 gibi sean-k-mooney: yeah, it is alway 61 seconds after the succesfull run of .ServersTestV219.test_description_errors that the unshelve test fails
11:48:51 gibi that sounds like a 60 sec timout on an RPC
11:49:02 songwenping_ thanks gibi. :(
11:49:16 gibi songwenping_: sorry I knee deep in someting else at the moment
11:50:02 sean-k-mooney songwenping_: nova does not move host between aggreates. you have to use the api to do that

Earlier   Later