| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-09 | |||
| 16:36:31 | dmitriis | but, yes, I think early views would be preferred | |
| 16:37:14 | bauzas | :) | |
| 16:37:31 | bauzas | ok, I guess we consumed the whole agenda | |
| 16:37:37 | dmitriis | there were some external dependencies in Libvirt and OVN that got merged recently (so this is out of the way). During the PTG we agreed that the Neutron spec needs to be reviewed first and that I need to address some additional points | |
| 16:37:39 | dmitriis | ack | |
| 16:37:50 | bauzas | dmitriis: yup, indeed | |
| 16:38:07 | bauzas | dmitriis: but yeah, I get the fact the dependencies are now solved | |
| 16:38:10 | bauzas | so it's our turn | |
| 16:39:04 | bauzas | anyone wanting to raise anything before we shutdown the meeting ? | |
| 16:40:11 | gibi | - | |
| 16:40:20 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.log.html | |
| 16:40:20 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.txt | |
| 16:40:20 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.html | |
| 16:40:20 | opendevmeet | Meeting ended Tue Nov 9 16:40:20 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:40:20 | bauzas | #endmeeting | |
| 16:40:24 | dmitriis | o/ | |
| 16:40:35 | elodilles | o/ | |
| 16:40:37 | bauzas | hah, fancy https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.html | |
| 16:41:32 | bauzas | I fixed the use of the #link command and the topics | |
| 16:42:02 | bauzas | I guess I need to make sure we provide an #info command per topîc | |
| 16:48:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Apply common irrelevant_files for centos 8 job https://review.opendev.org/c/openstack/nova/+/817278 | |
| 16:49:27 | gibi | lyarwood: ^^ on tweak for the new job | |
| 16:49:31 | gibi | *one | |
| 16:51:00 | lyarwood | gibi: it's part of tempest-integrated-compute so is this really needed? | |
| 16:51:18 | lyarwood | ah wait the template is called something different my bad | |
| 16:51:41 | gibi | yeah, I noticed that it was run on https://review.opendev.org/c/openstack/nova/+/814735 but no other tempest job run there | |
| 17:02:08 | opendevreview | Balazs Gibizer proposed openstack/placement stable/xena: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/817255 | |
| 17:02:10 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | |
| 17:02:41 | gibi | bauzas, gmann: ^^ these backports are needed to make the periodic placement test run green on stable/xena | |
| 17:02:58 | gibi | as per https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly | |
| 17:04:16 | opendevreview | Balazs Gibizer proposed openstack/placement stable/xena: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/817255 | |
| 17:05:34 | opendevreview | Stephen Finucane proposed openstack/nova master: Use unittest.mock instead of third party mock https://review.opendev.org/c/openstack/nova/+/714676 | |
| 17:15:38 | gibi | stephenfin: on the backport of the Database fixture fix, I think we need to backport https://review.opendev.org/c/openstack/nova/+/810291 as well | |
| 17:16:21 | gibi | or at least I see that as a difference between master and xena and my backport on xena now fails misteriously https://zuul.opendev.org/t/openstack/build/d7c064c8981b40618e3d24fc221c1832/log/job-output.txt | |
| 17:18:46 | gibi | anyhow I gave up for today | |
| 17:18:46 | kevko | anyone to help me investigate nova/neutron problem :/ | |
| 17:18:47 | sean-k-mooney | gibi: i have not reviewed that but skimmig it quickly it seam like a small enough change | |
| 17:19:34 | gibi | sean-k-mooney: me neither, I probably need to pull it in apply it to xena and see if it resolves the test failure with the xena backport | |
| 17:19:56 | kevko | sean-k-mooney: hi, i patched nova code to see how much time spent to get event about vif plugged from neutron | |
| 17:21:02 | kevko | on my test environment it is about 10 - 40 sec ..sometimes it is higher sometimes it is lower ..what is strange that sometimes when I run heat stack ..it is quite fast and I can see debug log message about vif event ..sometimes it is long time .. :( | |
| 17:24:47 | sean-k-mooney | kevko: it soundly like when there are a lot of vms strating it presumable gets longer | |
| 17:25:08 | sean-k-mooney | are you seeing them get close to the 300 time out or are they still generally below that | |
| 17:26:10 | kevko | sean-k-mooney: nope, it is really low | |
| 17:26:14 | kevko | sean-k-mooney: https://paste.opendev.org/show/810887/ | |
| 17:26:58 | kevko | stack is always same .. 6 small cirros instances | |
| 17:27:26 | kevko | openstack is clean testing env ..so no other processes running ...just my stack is building .. | |
| 17:28:20 | sean-k-mooney | kevko: that point to this not being a general performance problem then so increaseign the timeout wont help | |
| 17:28:44 | kevko | sean-k-mooney: yeah, something is somewhere buggy :D probably in neutron .. | |
| 17:28:52 | sean-k-mooney | you will have to start corralating the nova and netron logs to see if/when the ovs ports ar created and what happens | |
| 17:29:06 | kevko | on neutron-server side i can see this -> 2021-11-09 16:40:42.749 8 ERROR neutron.agent.dhcp.agent [-] Unexpected number of DHCP interfaces for metadata proxy, expected 1, got 2 | |
| 17:29:31 | sean-k-mooney | ya it either a problem with libvirt creating the tap and adding it to ovs or a proablem in the neutron l2 agent | |
| 17:30:28 | kevko | hmm, If you have a time ..I can give you access to that LAB env | |
| 17:30:47 | kevko | sean-k-mooney: or give logs ? | |
| 17:31:03 | kevko | sean-k-mooney: because I don't know if I am able to debug it :/ ..trying whole day | |
| 17:32:10 | sean-k-mooney | if you can share logs form 5-10min before/after the vm failed for the neutron l2 agent and nova-compute agent that should be enough | |
| 17:32:26 | sean-k-mooney | i can try and take a look but unfrotruatlly i proabley wont be able to fully debug this for you | |
| 17:33:04 | kevko | ok, give me minute | |
| 17:33:07 | sean-k-mooney | really the way to approch this is look for the point at which nova/libvirt create teh docmain which will in trun create teh port and get the time stampe | |
| 17:33:36 | sean-k-mooney | then you need to look at the l2 agent log and see if it start processign the port in the treat_ports fuction | |
| 17:34:38 | sean-k-mooney | kevko: this is the code that shoudl configure the port after its added https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1925 | |
| 17:37:30 | kevko | sean-k-mooney: https://debian.kevko.ultimum.cloud/neutron-openvswitch-agent.log | |
| 17:38:47 | sean-k-mooney | do you know the uuid of the port/tap name or mac | |
| 17:39:04 | kevko | nova-compute 17:16:53.188 line | |
| 17:40:21 | kevko | sean-k-mooney: probably this ? | |
| 17:40:22 | kevko | us_dns_name=''), 'device_id': '', 'bindings': [PortBinding(host='',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')], 'device_owner': ''} New fields: {'dns': PortDNS(current_dns_domain='',current_dns_name='',dns_domain='',dns_name='prod-p0000000001-s0000000001-uan',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,previous_dns_domain='',previous_dns_name= | |
| 17:40:22 | kevko | 2021-11-09 17:11:24.974 7 DEBUG neutron.agent.resource_cache [req-72e6674d-a4b6-4040-b62f-e7983c5c74f3 f21b4913a25d411fa774338091bd105a 5bd5561af79540c38df13222dce135f6 - - -] Resource Port 8d3373f0-6329-4252-8114-fc981873e0fb updated (revision_number 21->22). Old fields: {'dns': PortDNS(current_dns_domain='',current_dns_name='',dns_domain='',dns_name='',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,previous_dns_domain='',previo | |
| 17:40:24 | kevko | ''), 'device_id': 'f01680bd-ba12-4029-b11f-b2d5ae848818', 'bindings': [PortBinding(host='compute0',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')], 'device_owner': 'compute:nova'} record_resource_update /usr/lib/python3/dist-packages/neutron/agent/resource_cache.py:185 | |
| 17:40:29 | kevko | found via instance id | |
| 17:42:09 | sean-k-mooney | ok so tha tis in the log | |
| 17:42:11 | sean-k-mooney | 021-11-09 17:05:32.150 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-70183bba-0380-45d0-afef-7834c5644b2a - - - - -] Starting to process devices in:{'current': {'8d3373f0-6329-4252-8114-fc981873e0fb', 'f2b15696-b359-4216-a22a-804ebf285332', 'cc1609e0-d7ae-45a1-9405-1c95bb8dabf1', '05295e05-4fc2-4c00-ba77-8e4ff57b2ae3'}, 'added': set(), 'removed': | |
| 17:42:13 | sean-k-mooney | set(), 'updated': {'8d3373f0-6329-4252-8114-fc981873e0fb', 'f2b15696-b359-4216-a22a-804ebf285332', '05295e05-4fc2-4c00-ba77-8e4ff57b2ae3'}, 're_added': set()} rpc_loop /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2662 | |
| 17:43:48 | sean-k-mooney | and the status is set up at 17:05:41 | |
| 17:43:50 | sean-k-mooney | 2021-11-09 17:05:41.766 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-70183bba-0380-45d0-afef-7834c5644b2a - - - - -] Setting status for 8d3373f0-6329-4252-8114-fc981873e0fb to UP _bind_devices /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1202 | |
| 17:45:30 | sean-k-mooney | so the device configuration complete in the agent a 17:05:44 | |
| 17:46:22 | sean-k-mooney | a side effect fo settign the status up shoudl be calling the provision blocks cod ewhich will eventurally send the even to nova | |
| 17:50:39 | sean-k-mooney | kevko: that intersting im seeign it repeat later in the log too | |
| 17:51:32 | kevko | well, i think if something is broken ..it is trying to spawn instance again no ? | |
| 17:52:36 | sean-k-mooney | not on the same host | |
| 17:53:19 | sean-k-mooney | the curernt host appears to be compute0 | |
| 17:55:31 | sean-k-mooney | the revision_number 23->24 update is fefintly going form bound to compute0 with status down to compute0 with status up | |
| 17:56:30 | sean-k-mooney | which corralates with the 21->22 detail above | |
| 17:56:52 | sean-k-mooney | it looks like the issue is else where perhaps in the the dhcp agent or neutorn server | |
| 17:57:22 | sean-k-mooney | kevko: for the event to be sent both the l2 agnet and dhcp agent need to notify the neutron server that the provisioning is complete | |
| 17:58:22 | sean-k-mooney | since the l2 agent seams to be working correctly the next most likely candiate is the dhcp agent being slow whne many vms are created | |
| 17:58:36 | kevko | 6 vms ? :/ | |
| 18:01:16 | sean-k-mooney | its likely that there is a bug in the configutaion that is cause the agent to block/hang for some reason if this is the issue | |
| 18:01:36 | sean-k-mooney | its not really a perfroamce issue | |
| 18:01:54 | sean-k-mooney | we have had bugs in the interactio nwith dnsmasque in the past | |
| 18:02:33 | sean-k-mooney | kevko: in any case when the l2 agent set the port status as active it execution this code which mare it complete for the l2 agent | |
| 18:02:35 | sean-k-mooney | https://github.com/openstack/neutron/blob/9241c76b04e6745cc648ee42037cfe6ddad3600a/neutron/plugins/ml2/rpc.py#L312-L331 | |
| 18:02:50 | sean-k-mooney | if both sides had complted the provision the event would have been sent | |
| 18:03:45 | kevko | bug in configuration ? | |
| 18:04:08 | kevko | yeah, i saws some fixed bugs on launchpad | |
| 18:05:35 | sean-k-mooney | in the neutron server you should see one of these two logs notign that the l2 agent has complted its provisioning https://github.com/openstack/neutron/blob/9241c76b04e6745cc648ee42037cfe6ddad3600a/neutron/db/provisioning_blocks.py#L133-L140 | |
| 18:10:37 | kevko | sean-k-mooney: nothing, i have wallaby btw | |
| 18:11:30 | sean-k-mooney | i dont think this has changed much form wallaby to master | |
| 18:12:09 | sean-k-mooney | it might be best to take this to then neutron channel butit would seam for whateer reason that the port status chagne is not propagating to the nutron server then | |
| 18:12:44 | kevko | do you want ssh key to that lab ? | |
| 18:13:27 | sean-k-mooney | unfortunetly i have some other work i need to get done so im not sure i can really supprot debuging this much beyond what i have already done | |
| 18:14:20 | kevko | sean-k-mooney: ok, no problem, thank you very much ... | |