| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-09 | |||
| 16:29:13 | bauzas | no worries, we can punt this to next week | |
| 16:29:52 | bauzas | #topic Open discussion | |
| 16:29:59 | bauzas | Off-path Network Backends spec re-review https://review.opendev.org/c/openstack/nova-specs/+/787458 after addressing PTG comments (dmitriis) | |
| 16:30:10 | bauzas | dmitriis: around ? | |
| 16:30:14 | dmitriis | yep | |
| 16:31:41 | dmitriis | There is some progress on that, I am waiting for a second +2 (hopefully there will be some more feedback today). | |
| 16:31:41 | dmitriis | One of the asks during the PTG was that the Neutron cores review the Neutron spec: https://review.opendev.org/c/openstack/neutron-specs/+/788821/ | |
| 16:31:59 | dmitriis | I updated the spec with some of the points that were discussed during the PTG as well | |
| 16:32:11 | bauzas | \o/ | |
| 16:32:23 | bauzas | so I guess it's our turn ? | |
| 16:32:40 | dmitriis | That would be much appreciated :^) | |
| 16:32:49 | bauzas | ok, so just a ping for reviews ? :) | |
| 16:32:58 | bauzas | nothing you wanna discuss with the team by now ? | |
| 16:33:05 | bauzas | some open left question, maybe ? | |
| 16:33:13 | dmitriis | yes, just a ping for now | |
| 16:33:50 | dmitriis | trying to get some eyes on it early since we are getting closer to the spec freeze and holidays | |
| 16:34:42 | bauzas | dmitriis: I guess you saw we plan a spec review day ? | |
| 16:34:59 | bauzas | dmitriis: this doesn't mean we won't review your spec *before* | |
| 16:35:13 | bauzas | but we would appreciate if you could be around on this particular day | |
| 16:35:14 | dmitriis | bauzas: yes, I plan to be around for that and ready to address feedback | |
| 16:35:23 | bauzas | dmitriis: excellent, thanks | |
| 16:35:50 | bauzas | given the size of the spec, first runs of reviews will be needed before the spec review day | |
| 16:36:02 | bauzas | but this helps to know you'll be arouind | |
| 16:36:14 | dmitriis | bauzas: it had some rounds of reviews around April/May 2021 already | |
| 16:36:31 | dmitriis | but, yes, I think early views would be preferred | |
| 16:37:14 | bauzas | :) | |
| 16:37:31 | bauzas | ok, I guess we consumed the whole agenda | |
| 16:37:37 | dmitriis | there were some external dependencies in Libvirt and OVN that got merged recently (so this is out of the way). During the PTG we agreed that the Neutron spec needs to be reviewed first and that I need to address some additional points | |
| 16:37:39 | dmitriis | ack | |
| 16:37:50 | bauzas | dmitriis: yup, indeed | |
| 16:38:07 | bauzas | dmitriis: but yeah, I get the fact the dependencies are now solved | |
| 16:38:10 | bauzas | so it's our turn | |
| 16:39:04 | bauzas | anyone wanting to raise anything before we shutdown the meeting ? | |
| 16:40:11 | gibi | - | |
| 16:40:20 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.log.html | |
| 16:40:20 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.txt | |
| 16:40:20 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.html | |
| 16:40:20 | opendevmeet | Meeting ended Tue Nov 9 16:40:20 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:40:20 | bauzas | #endmeeting | |
| 16:40:24 | dmitriis | o/ | |
| 16:40:35 | elodilles | o/ | |
| 16:40:37 | bauzas | hah, fancy https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.html | |
| 16:41:32 | bauzas | I fixed the use of the #link command and the topics | |
| 16:42:02 | bauzas | I guess I need to make sure we provide an #info command per topîc | |
| 16:48:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Apply common irrelevant_files for centos 8 job https://review.opendev.org/c/openstack/nova/+/817278 | |
| 16:49:27 | gibi | lyarwood: ^^ on tweak for the new job | |
| 16:49:31 | gibi | *one | |
| 16:51:00 | lyarwood | gibi: it's part of tempest-integrated-compute so is this really needed? | |
| 16:51:18 | lyarwood | ah wait the template is called something different my bad | |
| 16:51:41 | gibi | yeah, I noticed that it was run on https://review.opendev.org/c/openstack/nova/+/814735 but no other tempest job run there | |
| 17:02:08 | opendevreview | Balazs Gibizer proposed openstack/placement stable/xena: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/817255 | |
| 17:02:10 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | |
| 17:02:41 | gibi | bauzas, gmann: ^^ these backports are needed to make the periodic placement test run green on stable/xena | |
| 17:02:58 | gibi | as per https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly | |
| 17:04:16 | opendevreview | Balazs Gibizer proposed openstack/placement stable/xena: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/817255 | |
| 17:05:34 | opendevreview | Stephen Finucane proposed openstack/nova master: Use unittest.mock instead of third party mock https://review.opendev.org/c/openstack/nova/+/714676 | |
| 17:15:38 | gibi | stephenfin: on the backport of the Database fixture fix, I think we need to backport https://review.opendev.org/c/openstack/nova/+/810291 as well | |
| 17:16:21 | gibi | or at least I see that as a difference between master and xena and my backport on xena now fails misteriously https://zuul.opendev.org/t/openstack/build/d7c064c8981b40618e3d24fc221c1832/log/job-output.txt | |
| 17:18:46 | gibi | anyhow I gave up for today | |
| 17:18:46 | kevko | anyone to help me investigate nova/neutron problem :/ | |
| 17:18:47 | sean-k-mooney | gibi: i have not reviewed that but skimmig it quickly it seam like a small enough change | |
| 17:19:34 | gibi | sean-k-mooney: me neither, I probably need to pull it in apply it to xena and see if it resolves the test failure with the xena backport | |
| 17:19:56 | kevko | sean-k-mooney: hi, i patched nova code to see how much time spent to get event about vif plugged from neutron | |
| 17:21:02 | kevko | on my test environment it is about 10 - 40 sec ..sometimes it is higher sometimes it is lower ..what is strange that sometimes when I run heat stack ..it is quite fast and I can see debug log message about vif event ..sometimes it is long time .. :( | |
| 17:24:47 | sean-k-mooney | kevko: it soundly like when there are a lot of vms strating it presumable gets longer | |
| 17:25:08 | sean-k-mooney | are you seeing them get close to the 300 time out or are they still generally below that | |
| 17:26:10 | kevko | sean-k-mooney: nope, it is really low | |
| 17:26:14 | kevko | sean-k-mooney: https://paste.opendev.org/show/810887/ | |
| 17:26:58 | kevko | stack is always same .. 6 small cirros instances | |
| 17:27:26 | kevko | openstack is clean testing env ..so no other processes running ...just my stack is building .. | |
| 17:28:20 | sean-k-mooney | kevko: that point to this not being a general performance problem then so increaseign the timeout wont help | |
| 17:28:44 | kevko | sean-k-mooney: yeah, something is somewhere buggy :D probably in neutron .. | |
| 17:28:52 | sean-k-mooney | you will have to start corralating the nova and netron logs to see if/when the ovs ports ar created and what happens | |
| 17:29:06 | kevko | on neutron-server side i can see this -> 2021-11-09 16:40:42.749 8 ERROR neutron.agent.dhcp.agent [-] Unexpected number of DHCP interfaces for metadata proxy, expected 1, got 2 | |
| 17:29:31 | sean-k-mooney | ya it either a problem with libvirt creating the tap and adding it to ovs or a proablem in the neutron l2 agent | |
| 17:30:28 | kevko | hmm, If you have a time ..I can give you access to that LAB env | |
| 17:30:47 | kevko | sean-k-mooney: or give logs ? | |
| 17:31:03 | kevko | sean-k-mooney: because I don't know if I am able to debug it :/ ..trying whole day | |
| 17:32:10 | sean-k-mooney | if you can share logs form 5-10min before/after the vm failed for the neutron l2 agent and nova-compute agent that should be enough | |
| 17:32:26 | sean-k-mooney | i can try and take a look but unfrotruatlly i proabley wont be able to fully debug this for you | |
| 17:33:04 | kevko | ok, give me minute | |
| 17:33:07 | sean-k-mooney | really the way to approch this is look for the point at which nova/libvirt create teh docmain which will in trun create teh port and get the time stampe | |
| 17:33:36 | sean-k-mooney | then you need to look at the l2 agent log and see if it start processign the port in the treat_ports fuction | |
| 17:34:38 | sean-k-mooney | kevko: this is the code that shoudl configure the port after its added https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1925 | |
| 17:37:30 | kevko | sean-k-mooney: https://debian.kevko.ultimum.cloud/neutron-openvswitch-agent.log | |
| 17:38:47 | sean-k-mooney | do you know the uuid of the port/tap name or mac | |
| 17:39:04 | kevko | nova-compute 17:16:53.188 line | |
| 17:40:21 | kevko | sean-k-mooney: probably this ? | |
| 17:40:22 | kevko | us_dns_name=''), 'device_id': '', 'bindings': [PortBinding(host='',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')], 'device_owner': ''} New fields: {'dns': PortDNS(current_dns_domain='',current_dns_name='',dns_domain='',dns_name='prod-p0000000001-s0000000001-uan',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,previous_dns_domain='',previous_dns_name= | |
| 17:40:22 | kevko | 2021-11-09 17:11:24.974 7 DEBUG neutron.agent.resource_cache [req-72e6674d-a4b6-4040-b62f-e7983c5c74f3 f21b4913a25d411fa774338091bd105a 5bd5561af79540c38df13222dce135f6 - - -] Resource Port 8d3373f0-6329-4252-8114-fc981873e0fb updated (revision_number 21->22). Old fields: {'dns': PortDNS(current_dns_domain='',current_dns_name='',dns_domain='',dns_name='',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,previous_dns_domain='',previo | |
| 17:40:24 | kevko | ''), 'device_id': 'f01680bd-ba12-4029-b11f-b2d5ae848818', 'bindings': [PortBinding(host='compute0',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')], 'device_owner': 'compute:nova'} record_resource_update /usr/lib/python3/dist-packages/neutron/agent/resource_cache.py:185 | |
| 17:40:29 | kevko | found via instance id | |
| 17:42:09 | sean-k-mooney | ok so tha tis in the log | |
| 17:42:11 | sean-k-mooney | 021-11-09 17:05:32.150 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-70183bba-0380-45d0-afef-7834c5644b2a - - - - -] Starting to process devices in:{'current': {'8d3373f0-6329-4252-8114-fc981873e0fb', 'f2b15696-b359-4216-a22a-804ebf285332', 'cc1609e0-d7ae-45a1-9405-1c95bb8dabf1', '05295e05-4fc2-4c00-ba77-8e4ff57b2ae3'}, 'added': set(), 'removed': | |
| 17:42:13 | sean-k-mooney | set(), 'updated': {'8d3373f0-6329-4252-8114-fc981873e0fb', 'f2b15696-b359-4216-a22a-804ebf285332', '05295e05-4fc2-4c00-ba77-8e4ff57b2ae3'}, 're_added': set()} rpc_loop /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2662 | |
| 17:43:48 | sean-k-mooney | and the status is set up at 17:05:41 | |
| 17:43:50 | sean-k-mooney | 2021-11-09 17:05:41.766 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-70183bba-0380-45d0-afef-7834c5644b2a - - - - -] Setting status for 8d3373f0-6329-4252-8114-fc981873e0fb to UP _bind_devices /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1202 | |
| 17:45:30 | sean-k-mooney | so the device configuration complete in the agent a 17:05:44 | |
| 17:46:22 | sean-k-mooney | a side effect fo settign the status up shoudl be calling the provision blocks cod ewhich will eventurally send the even to nova | |
| 17:50:39 | sean-k-mooney | kevko: that intersting im seeign it repeat later in the log too | |