| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-09 | |||
| 16:38:07 | bauzas | dmitriis: but yeah, I get the fact the dependencies are now solved | |
| 16:38:10 | bauzas | so it's our turn | |
| 16:39:04 | bauzas | anyone wanting to raise anything before we shutdown the meeting ? | |
| 16:40:11 | gibi | - | |
| 16:40:20 | bauzas | #endmeeting | |
| 16:40:20 | opendevmeet | Meeting ended Tue Nov 9 16:40:20 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:40:20 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.html | |
| 16:40:20 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.txt | |
| 16:40:20 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.log.html | |
| 16:40:24 | dmitriis | o/ | |
| 16:40:35 | elodilles | o/ | |
| 16:40:37 | bauzas | hah, fancy https://meetings.opendev.org/meetings/nova/2021/nova.2021-11-09-16.01.html | |
| 16:41:32 | bauzas | I fixed the use of the #link command and the topics | |
| 16:42:02 | bauzas | I guess I need to make sure we provide an #info command per topîc | |
| 16:48:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Apply common irrelevant_files for centos 8 job https://review.opendev.org/c/openstack/nova/+/817278 | |
| 16:49:27 | gibi | lyarwood: ^^ on tweak for the new job | |
| 16:49:31 | gibi | *one | |
| 16:51:00 | lyarwood | gibi: it's part of tempest-integrated-compute so is this really needed? | |
| 16:51:18 | lyarwood | ah wait the template is called something different my bad | |
| 16:51:41 | gibi | yeah, I noticed that it was run on https://review.opendev.org/c/openstack/nova/+/814735 but no other tempest job run there | |
| 17:02:08 | opendevreview | Balazs Gibizer proposed openstack/placement stable/xena: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/817255 | |
| 17:02:10 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | |
| 17:02:41 | gibi | bauzas, gmann: ^^ these backports are needed to make the periodic placement test run green on stable/xena | |
| 17:02:58 | gibi | as per https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly | |
| 17:04:16 | opendevreview | Balazs Gibizer proposed openstack/placement stable/xena: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/817255 | |
| 17:05:34 | opendevreview | Stephen Finucane proposed openstack/nova master: Use unittest.mock instead of third party mock https://review.opendev.org/c/openstack/nova/+/714676 | |
| 17:15:38 | gibi | stephenfin: on the backport of the Database fixture fix, I think we need to backport https://review.opendev.org/c/openstack/nova/+/810291 as well | |
| 17:16:21 | gibi | or at least I see that as a difference between master and xena and my backport on xena now fails misteriously https://zuul.opendev.org/t/openstack/build/d7c064c8981b40618e3d24fc221c1832/log/job-output.txt | |
| 17:18:46 | kevko | anyone to help me investigate nova/neutron problem :/ | |
| 17:18:46 | gibi | anyhow I gave up for today | |
| 17:18:47 | sean-k-mooney | gibi: i have not reviewed that but skimmig it quickly it seam like a small enough change | |
| 17:19:34 | gibi | sean-k-mooney: me neither, I probably need to pull it in apply it to xena and see if it resolves the test failure with the xena backport | |
| 17:19:56 | kevko | sean-k-mooney: hi, i patched nova code to see how much time spent to get event about vif plugged from neutron | |
| 17:21:02 | kevko | on my test environment it is about 10 - 40 sec ..sometimes it is higher sometimes it is lower ..what is strange that sometimes when I run heat stack ..it is quite fast and I can see debug log message about vif event ..sometimes it is long time .. :( | |
| 17:24:47 | sean-k-mooney | kevko: it soundly like when there are a lot of vms strating it presumable gets longer | |
| 17:25:08 | sean-k-mooney | are you seeing them get close to the 300 time out or are they still generally below that | |
| 17:26:10 | kevko | sean-k-mooney: nope, it is really low | |
| 17:26:14 | kevko | sean-k-mooney: https://paste.opendev.org/show/810887/ | |
| 17:26:58 | kevko | stack is always same .. 6 small cirros instances | |
| 17:27:26 | kevko | openstack is clean testing env ..so no other processes running ...just my stack is building .. | |
| 17:28:20 | sean-k-mooney | kevko: that point to this not being a general performance problem then so increaseign the timeout wont help | |
| 17:28:44 | kevko | sean-k-mooney: yeah, something is somewhere buggy :D probably in neutron .. | |
| 17:28:52 | sean-k-mooney | you will have to start corralating the nova and netron logs to see if/when the ovs ports ar created and what happens | |
| 17:29:06 | kevko | on neutron-server side i can see this -> 2021-11-09 16:40:42.749 8 ERROR neutron.agent.dhcp.agent [-] Unexpected number of DHCP interfaces for metadata proxy, expected 1, got 2 | |
| 17:29:31 | sean-k-mooney | ya it either a problem with libvirt creating the tap and adding it to ovs or a proablem in the neutron l2 agent | |
| 17:30:28 | kevko | hmm, If you have a time ..I can give you access to that LAB env | |
| 17:30:47 | kevko | sean-k-mooney: or give logs ? | |
| 17:31:03 | kevko | sean-k-mooney: because I don't know if I am able to debug it :/ ..trying whole day | |
| 17:32:10 | sean-k-mooney | if you can share logs form 5-10min before/after the vm failed for the neutron l2 agent and nova-compute agent that should be enough | |
| 17:32:26 | sean-k-mooney | i can try and take a look but unfrotruatlly i proabley wont be able to fully debug this for you | |
| 17:33:04 | kevko | ok, give me minute | |
| 17:33:07 | sean-k-mooney | really the way to approch this is look for the point at which nova/libvirt create teh docmain which will in trun create teh port and get the time stampe | |
| 17:33:36 | sean-k-mooney | then you need to look at the l2 agent log and see if it start processign the port in the treat_ports fuction | |
| 17:34:38 | sean-k-mooney | kevko: this is the code that shoudl configure the port after its added https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1925 | |
| 17:37:30 | kevko | sean-k-mooney: https://debian.kevko.ultimum.cloud/neutron-openvswitch-agent.log | |
| 17:38:47 | sean-k-mooney | do you know the uuid of the port/tap name or mac | |
| 17:39:04 | kevko | nova-compute 17:16:53.188 line | |
| 17:40:21 | kevko | sean-k-mooney: probably this ? | |
| 17:40:22 | kevko | 2021-11-09 17:11:24.974 7 DEBUG neutron.agent.resource_cache [req-72e6674d-a4b6-4040-b62f-e7983c5c74f3 f21b4913a25d411fa774338091bd105a 5bd5561af79540c38df13222dce135f6 - - -] Resource Port 8d3373f0-6329-4252-8114-fc981873e0fb updated (revision_number 21->22). Old fields: {'dns': PortDNS(current_dns_domain='',current_dns_name='',dns_domain='',dns_name='',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,previous_dns_domain='',previo | |
| 17:40:22 | kevko | us_dns_name=''), 'device_id': '', 'bindings': [PortBinding(host='',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')], 'device_owner': ''} New fields: {'dns': PortDNS(current_dns_domain='',current_dns_name='',dns_domain='',dns_name='prod-p0000000001-s0000000001-uan',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,previous_dns_domain='',previous_dns_name= | |
| 17:40:24 | kevko | ''), 'device_id': 'f01680bd-ba12-4029-b11f-b2d5ae848818', 'bindings': [PortBinding(host='compute0',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')], 'device_owner': 'compute:nova'} record_resource_update /usr/lib/python3/dist-packages/neutron/agent/resource_cache.py:185 | |
| 17:40:29 | kevko | found via instance id | |
| 17:42:09 | sean-k-mooney | ok so tha tis in the log | |
| 17:42:11 | sean-k-mooney | 021-11-09 17:05:32.150 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-70183bba-0380-45d0-afef-7834c5644b2a - - - - -] Starting to process devices in:{'current': {'8d3373f0-6329-4252-8114-fc981873e0fb', 'f2b15696-b359-4216-a22a-804ebf285332', 'cc1609e0-d7ae-45a1-9405-1c95bb8dabf1', '05295e05-4fc2-4c00-ba77-8e4ff57b2ae3'}, 'added': set(), 'removed': | |
| 17:42:13 | sean-k-mooney | set(), 'updated': {'8d3373f0-6329-4252-8114-fc981873e0fb', 'f2b15696-b359-4216-a22a-804ebf285332', '05295e05-4fc2-4c00-ba77-8e4ff57b2ae3'}, 're_added': set()} rpc_loop /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2662 | |
| 17:43:48 | sean-k-mooney | and the status is set up at 17:05:41 | |
| 17:43:50 | sean-k-mooney | 2021-11-09 17:05:41.766 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-70183bba-0380-45d0-afef-7834c5644b2a - - - - -] Setting status for 8d3373f0-6329-4252-8114-fc981873e0fb to UP _bind_devices /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1202 | |
| 17:45:30 | sean-k-mooney | so the device configuration complete in the agent a 17:05:44 | |
| 17:46:22 | sean-k-mooney | a side effect fo settign the status up shoudl be calling the provision blocks cod ewhich will eventurally send the even to nova | |
| 17:50:39 | sean-k-mooney | kevko: that intersting im seeign it repeat later in the log too | |
| 17:51:32 | kevko | well, i think if something is broken ..it is trying to spawn instance again no ? | |
| 17:52:36 | sean-k-mooney | not on the same host | |
| 17:53:19 | sean-k-mooney | the curernt host appears to be compute0 | |
| 17:55:31 | sean-k-mooney | the revision_number 23->24 update is fefintly going form bound to compute0 with status down to compute0 with status up | |
| 17:56:30 | sean-k-mooney | which corralates with the 21->22 detail above | |
| 17:56:52 | sean-k-mooney | it looks like the issue is else where perhaps in the the dhcp agent or neutorn server | |
| 17:57:22 | sean-k-mooney | kevko: for the event to be sent both the l2 agnet and dhcp agent need to notify the neutron server that the provisioning is complete | |
| 17:58:22 | sean-k-mooney | since the l2 agent seams to be working correctly the next most likely candiate is the dhcp agent being slow whne many vms are created | |
| 17:58:36 | kevko | 6 vms ? :/ | |
| 18:01:16 | sean-k-mooney | its likely that there is a bug in the configutaion that is cause the agent to block/hang for some reason if this is the issue | |
| 18:01:36 | sean-k-mooney | its not really a perfroamce issue | |
| 18:01:54 | sean-k-mooney | we have had bugs in the interactio nwith dnsmasque in the past | |
| 18:02:33 | sean-k-mooney | kevko: in any case when the l2 agent set the port status as active it execution this code which mare it complete for the l2 agent | |
| 18:02:35 | sean-k-mooney | https://github.com/openstack/neutron/blob/9241c76b04e6745cc648ee42037cfe6ddad3600a/neutron/plugins/ml2/rpc.py#L312-L331 | |
| 18:02:50 | sean-k-mooney | if both sides had complted the provision the event would have been sent | |
| 18:03:45 | kevko | bug in configuration ? | |
| 18:04:08 | kevko | yeah, i saws some fixed bugs on launchpad | |
| 18:05:35 | sean-k-mooney | in the neutron server you should see one of these two logs notign that the l2 agent has complted its provisioning https://github.com/openstack/neutron/blob/9241c76b04e6745cc648ee42037cfe6ddad3600a/neutron/db/provisioning_blocks.py#L133-L140 | |
| 18:10:37 | kevko | sean-k-mooney: nothing, i have wallaby btw | |
| 18:11:30 | sean-k-mooney | i dont think this has changed much form wallaby to master | |
| 18:12:09 | sean-k-mooney | it might be best to take this to then neutron channel butit would seam for whateer reason that the port status chagne is not propagating to the nutron server then | |
| 18:12:44 | kevko | do you want ssh key to that lab ? | |
| 18:13:27 | sean-k-mooney | unfortunetly i have some other work i need to get done so im not sure i can really supprot debuging this much beyond what i have already done | |
| 18:14:20 | kevko | sean-k-mooney: ok, no problem, thank you very much ... | |
| 18:14:41 | kevko | btw, I have neutron server set to Debug = False ..so that's the reason why I am not seeing that debug messages .. | |
| 18:15:04 | sean-k-mooney | ah ya these are debug only since its a bit verbose | |
| 18:16:06 | kevko | ok, have to go ...thank you very much | |
| 20:16:21 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Run OVS job with hybrid plug https://review.opendev.org/c/openstack/nova/+/817303 | |
| 20:18:47 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Run OVS job with hybrid plug https://review.opendev.org/c/openstack/nova/+/817303 | |
| 20:26:20 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Revert project-specific APIs for servers https://review.opendev.org/c/openstack/nova/+/816206 | |