Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-09
17:30:28 kevko hmm, If you have a time ..I can give you access to that LAB env
17:30:47 kevko sean-k-mooney: or give logs ?
17:31:03 kevko sean-k-mooney: because I don't know if I am able to debug it :/ ..trying whole day
17:32:10 sean-k-mooney if you can share logs form 5-10min before/after the vm failed for the neutron l2 agent and nova-compute agent that should be enough
17:32:26 sean-k-mooney i can try and take a look but unfrotruatlly i proabley wont be able to fully debug this for you
17:33:04 kevko ok, give me minute
17:33:07 sean-k-mooney really the way to approch this is look for the point at which nova/libvirt create teh docmain which will in trun create teh port and get the time stampe
17:33:36 sean-k-mooney then you need to look at the l2 agent log and see if it start processign the port in the treat_ports fuction
17:34:38 sean-k-mooney kevko: this is the code that shoudl configure the port after its added https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1925
17:37:30 kevko sean-k-mooney: https://debian.kevko.ultimum.cloud/neutron-openvswitch-agent.log
17:38:47 sean-k-mooney do you know the uuid of the port/tap name or mac
17:39:04 kevko nova-compute 17:16:53.188 line
17:40:21 kevko sean-k-mooney: probably this ?
17:40:22 kevko us_dns_name=''), 'device_id': '', 'bindings': [PortBinding(host='',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')], 'device_owner': ''} New fields: {'dns': PortDNS(current_dns_domain='',current_dns_name='',dns_domain='',dns_name='prod-p0000000001-s0000000001-uan',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,previous_dns_domain='',previous_dns_name=
17:40:22 kevko 2021-11-09 17:11:24.974 7 DEBUG neutron.agent.resource_cache [req-72e6674d-a4b6-4040-b62f-e7983c5c74f3 f21b4913a25d411fa774338091bd105a 5bd5561af79540c38df13222dce135f6 - - -] Resource Port 8d3373f0-6329-4252-8114-fc981873e0fb updated (revision_number 21->22). Old fields: {'dns': PortDNS(current_dns_domain='',current_dns_name='',dns_domain='',dns_name='',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,previous_dns_domain='',previo
17:40:24 kevko ''), 'device_id': 'f01680bd-ba12-4029-b11f-b2d5ae848818', 'bindings': [PortBinding(host='compute0',port_id=8d3373f0-6329-4252-8114-fc981873e0fb,profile={},status='ACTIVE',vif_details=None,vif_type='unbound',vnic_type='normal')], 'device_owner': 'compute:nova'} record_resource_update /usr/lib/python3/dist-packages/neutron/agent/resource_cache.py:185
17:40:29 kevko found via instance id
17:42:09 sean-k-mooney ok so tha tis in the log
17:42:11 sean-k-mooney 021-11-09 17:05:32.150 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-70183bba-0380-45d0-afef-7834c5644b2a - - - - -] Starting to process devices in:{'current': {'8d3373f0-6329-4252-8114-fc981873e0fb', 'f2b15696-b359-4216-a22a-804ebf285332', 'cc1609e0-d7ae-45a1-9405-1c95bb8dabf1', '05295e05-4fc2-4c00-ba77-8e4ff57b2ae3'}, 'added': set(), 'removed':
17:42:13 sean-k-mooney set(), 'updated': {'8d3373f0-6329-4252-8114-fc981873e0fb', 'f2b15696-b359-4216-a22a-804ebf285332', '05295e05-4fc2-4c00-ba77-8e4ff57b2ae3'}, 're_added': set()} rpc_loop /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2662
17:43:48 sean-k-mooney and the status is set up at 17:05:41
17:43:50 sean-k-mooney 2021-11-09 17:05:41.766 7 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-70183bba-0380-45d0-afef-7834c5644b2a - - - - -] Setting status for 8d3373f0-6329-4252-8114-fc981873e0fb to UP _bind_devices /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1202
17:45:30 sean-k-mooney so the device configuration complete in the agent a 17:05:44
17:46:22 sean-k-mooney a side effect fo settign the status up shoudl be calling the provision blocks cod ewhich will eventurally send the even to nova
17:50:39 sean-k-mooney kevko: that intersting im seeign it repeat later in the log too
17:51:32 kevko well, i think if something is broken ..it is trying to spawn instance again no ?
17:52:36 sean-k-mooney not on the same host
17:53:19 sean-k-mooney the curernt host appears to be compute0
17:55:31 sean-k-mooney the revision_number 23->24 update is fefintly going form bound to compute0 with status down to compute0 with status up
17:56:30 sean-k-mooney which corralates with the 21->22 detail above
17:56:52 sean-k-mooney it looks like the issue is else where perhaps in the the dhcp agent or neutorn server
17:57:22 sean-k-mooney kevko: for the event to be sent both the l2 agnet and dhcp agent need to notify the neutron server that the provisioning is complete
17:58:22 sean-k-mooney since the l2 agent seams to be working correctly the next most likely candiate is the dhcp agent being slow whne many vms are created
17:58:36 kevko 6 vms ? :/
18:01:16 sean-k-mooney its likely that there is a bug in the configutaion that is cause the agent to block/hang for some reason if this is the issue
18:01:36 sean-k-mooney its not really a perfroamce issue
18:01:54 sean-k-mooney we have had bugs in the interactio nwith dnsmasque in the past
18:02:33 sean-k-mooney kevko: in any case when the l2 agent set the port status as active it execution this code which mare it complete for the l2 agent
18:02:35 sean-k-mooney https://github.com/openstack/neutron/blob/9241c76b04e6745cc648ee42037cfe6ddad3600a/neutron/plugins/ml2/rpc.py#L312-L331
18:02:50 sean-k-mooney if both sides had complted the provision the event would have been sent
18:03:45 kevko bug in configuration ?
18:04:08 kevko yeah, i saws some fixed bugs on launchpad
18:05:35 sean-k-mooney in the neutron server you should see one of these two logs notign that the l2 agent has complted its provisioning https://github.com/openstack/neutron/blob/9241c76b04e6745cc648ee42037cfe6ddad3600a/neutron/db/provisioning_blocks.py#L133-L140
18:10:37 kevko sean-k-mooney: nothing, i have wallaby btw
18:11:30 sean-k-mooney i dont think this has changed much form wallaby to master
18:12:09 sean-k-mooney it might be best to take this to then neutron channel butit would seam for whateer reason that the port status chagne is not propagating to the nutron server then
18:12:44 kevko do you want ssh key to that lab ?
18:13:27 sean-k-mooney unfortunetly i have some other work i need to get done so im not sure i can really supprot debuging this much beyond what i have already done
18:14:20 kevko sean-k-mooney: ok, no problem, thank you very much ...
18:14:41 kevko btw, I have neutron server set to Debug = False ..so that's the reason why I am not seeing that debug messages ..
18:15:04 sean-k-mooney ah ya these are debug only since its a bit verbose
18:16:06 kevko ok, have to go ...thank you very much
20:16:21 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: Run OVS job with hybrid plug https://review.opendev.org/c/openstack/nova/+/817303
20:18:47 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: Run OVS job with hybrid plug https://review.opendev.org/c/openstack/nova/+/817303
20:26:20 opendevreview Dan Smith proposed openstack/nova master: WIP: Revert project-specific APIs for servers https://review.opendev.org/c/openstack/nova/+/816206
20:26:42 dansmith gmann: lbragstad: my brain is fried from ^ so use extra caution while reviewing
20:27:06 dansmith however, I do think that's much easier to read than what was there before, and hopefully makes the iteration from current..scope..nolegacy more clear
20:35:12 gmann dansmith: thanks, ack
20:45:40 lbragstad dansmith sweet - thanks
21:06:22 hyang[m] Hi there, can someone help to review https://review.opendev.org/c/openstack/nova/+/811521? It can help to close both https://bugs.launchpad.net/nova/+bug/1943969 and https://bugs.launchpad.net/neutron/+bug/1942615
22:14:57 artom Hah, so revert resize is broken with ovs + hybrid plug
22:15:01 artom https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f60/817303/2/check/nova-ovs-hybrid-plug/f60d54c/testr_results.html
22:15:13 artom We first noticed this downstream, and now that ^^ tested it upstream, same result
22:19:18 gmann dansmith: lbragstad johnthetubaguy[m] I created this wikitable to audit all the nova API policy - https://wiki.openstack.org/wiki/Nova/rbac
22:20:20 gmann few I have kept as ? mainly multi-policy one. for example showing host_status policy in GET /servers please review those.
22:24:38 gmann dansmith: lbragstad johnthetubaguy[m] I have updated those as per my understanding and with new direction we agreed on Wed. My eyes are paining now after listing/auditing these ~225 policies . will catch up on this tomorrow.
22:32:09 dansmith gmann: wow, I thought you were going to do it in a google sheet or something
22:32:23 dansmith I'm sure your eyes are literally bleeding now :/
#openstack-nova - 2021-11-10
02:19:15 opendevreview Wenping Song proposed openstack/nova master: Fill the AcceleratorRequestBindingFailed exception msg info https://review.opendev.org/c/openstack/nova/+/817326
02:52:28 opendevreview Wenping Song proposed openstack/nova master: Fill the AcceleratorRequestBindingFailed exception msg info https://review.opendev.org/c/openstack/nova/+/817326
09:09:22 opendevreview likui proposed openstack/osc-placement master: Replace deprecated assertRaisesRegexp https://review.opendev.org/c/openstack/osc-placement/+/817365
09:52:04 stephenfin lyarwood: the py36 tox environment is failing for me locally (F34). I proposed this to fix it https://github.com/powervm/pypowervm/pull/17
09:53:18 stephenfin We'll need that to merge and a new release of pypowervm be cut before we can get things running again. I suspect we're not seeing issues in the gate yet because an older dependency resolver that doesn't care about these things is being used there
09:53:26 stephenfin Just FYI since you're also using Fedora :)
10:07:46 gibi elodilles, lyarwood: hi! re: kernel panic in stable/victoria with volume detach: https://bugs.launchpad.net/nova/+bug/1950310 yesterday I backported the libvirt event based detach series to victoria but top of that series the kernel panic still happens (links in the bug for the recent failures)
10:08:02 gibi I'm out of ideas now
10:24:44 sean-k-mooney[m] gibi i could maybe try to revive my attempts to add support for alpine to dib so that we could use it instead of cirros
10:25:33 sean-k-mooney[m] i started that orignally due to kernel bugs in the cirros kernel
10:26:27 gibi sean-k-mooney[m]: I'm not sure if it is really cirros. i) it happens with both cirros 0.5.1 and 0.5.2 but it does not happen on master just on stable/victoria
10:26:42 gibi s/i)//
10:26:52 sean-k-mooney[m] i see
10:27:52 sean-k-mooney[m] if its still happening with you event based detach patch the only other thing i can think of is changing the qemu version we are using
10:29:11 sean-k-mooney[m] i used to maintain a devstack plugin to compile qemu form git but i have not updated it to support the mason build systme wich they now use. we could try changing the qemu we use either via a ppa or compiling it
10:29:16 gibi hm interesting, I don't know how different the qemu version between master and stable/victoria but I can check that
10:29:40 sean-k-mooney[m] master would be focal is stable victoria on bionic?
10:33:14 frickler gibi: sean-k-mooney[m]: if you want newer qemu, could you try running on bullseye instead ubuntu?
10:34:53 sean-k-mooney[m] frickler: bullseye and centos would be options. its not really about newer it could be older. if the panics are caused by a kvm or qemu bug then just using a diffent disto with different verions might fix the problem
10:36:06 lyarwood gibi: yeah I didn't think it would help tbh, you're still firing device_del down to QEMU and the ACPI requests to the guestOS, are we pinning tempest on stable/victoria? There was an effort recently to force tempest to wait until the guestOS was up before attaching and detaching volumes, I wonder if that logic is there?
10:36:43 lyarwood gibi: I've had my head downstream on something since yesterday but I'll try to help out some more for the rest of the week
10:37:27 sean-k-mooney[m] you mean adding the validation so that we ssh into the vm before atching/detaching
10:37:33 sean-k-mooney[m] or something else
10:37:55 lyarwood sean-k-mooney: yeah that's it
10:38:09 lyarwood stephenfin: ack not seen that personally but will keep it in mind
10:39:02 sean-k-mooney[m] lyarwood has anyone submited patches for that
10:39:20 lyarwood I had thought it already landed in master tbh
10:41:01 lyarwood okay no it hasn't
10:41:25 lyarwood I was sure I saw a change land for this or maybe my dreams are really becoming boring now I'm half way through my 30's
10:42:45 sean-k-mooney[m] we spoke about it in out inernal meeting last week but unless you wrote it i dont think it happened.
10:43:19 sean-k-mooney[m] if i remember correctly we just need to set validation=true on the create server line in the test
11:30:33 gibi lyarwood: do you have a link for the patch that adds that validation? I can add a depends-on somewhere in nova to see if that helps

Earlier   Later