Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-06-30
14:34:37 stephenfin done
14:36:44 sean-k-mooney many thanks
14:37:56 opendevreview Merged openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545
14:56:28 stephenfin sean-k-mooney: Seen this before? https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-n-cpu.txt#10060
14:56:34 stephenfin nova.exception.InternalError: Unexpected vif_type=unbound
14:56:46 stephenfin It rings a bell, but I thought we'd fixed this years ago
14:57:09 sean-k-mooney we had older bugs related to unbound
14:57:22 sean-k-mooney that is the state when the host-id is not set on the port
14:57:52 sean-k-mooney i think i looked at this
14:58:21 sean-k-mooney No conversion for VIF type unbound yet {{(pid=97953) nova_to_osvif_vif /opt/stack/nova/nova/network/os_vif_util.py:530}}
14:58:37 sean-k-mooney is really just a side effect of the prot not being bound properly in neutron
15:00:35 sean-k-mooney ah right
15:00:37 sean-k-mooney https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10891
15:00:50 sean-k-mooney so this is an issue with slow neutron i think
15:00:52 opendevreview Merged openstack/nova stable/wallaby: fake: Ensure need_legacy_block_device_info returns False https://review.opendev.org/c/openstack/nova/+/843678
15:00:59 opendevreview Merged openstack/nova stable/wallaby: Add a regression test for bug 1939545 https://review.opendev.org/c/openstack/nova/+/843679
15:01:22 sean-k-mooney stephenfin: basically when i was looking at this i was assumeing this happened because we retried the port bidning because neutron was slow to responed
15:01:44 sean-k-mooney and that caused a concurnet bind attempt that left it in ubound or something like that
15:02:10 stephenfin Hmm, that sounds reasonable. We saw it bubble up in the OSC tests because the server create failed. That sounds like a likely root cause though
15:02:46 sean-k-mooney if that is what is happening i woudl expect to see the retry logged somewher or see two bind attempts in the nutron log
15:02:54 sean-k-mooney ill check if i can see that
15:03:28 sean-k-mooney we do see the port had just finished binding when we got the concurrent error
15:03:29 sean-k-mooney ] Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id
15:05:10 sean-k-mooney ya it looks like there are 3 attempts to bind the port
15:05:23 frickler stephenfin: I mentioned that yesterday, it also seemed related to neutron retrying binds
15:05:26 sean-k-mooney in the neutron side the last to of which had the concurrent bind excption
15:05:45 sean-k-mooney frickler: well its actully the neutornclient retrying the bind
15:06:02 sean-k-mooney frickler: that was entirely broken in nova until somewhat recently
15:06:11 melwitt bauzas: I wanted to get your thoughts on this proposed patch to change logic in the placement audit nova-manage command, since you worked on it https://review.opendev.org/c/openstack/nova/+/844418 it seems like there is a bug in the current logic but it's not clear to me what the logic should be
15:06:23 sean-k-mooney frickler: we fixed retires about a year or so ago
15:06:30 bauzas melwitt: okay, I'll look
15:06:37 melwitt thanks
15:06:41 frickler sean-k-mooney: iiuc there is an internal retry in neutron happening now
15:07:04 sean-k-mooney frickler: there is also likely one in the db decorator
15:07:20 sean-k-mooney we can see transaction error in the log
15:07:42 frickler is this with neutron-segment enabled? we have some issue with segment ID reuse in OSC
15:08:34 sean-k-mooney i think segments were enabled yes
15:08:45 frickler oh, that's the osc job, yes
15:09:28 sean-k-mooney its using vxlan however i think
15:09:39 sean-k-mooney rather then routed provider networks
15:10:07 sean-k-mooney Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id': 'cd5c5c6b-1027-4fc7-bbc7-b8204df12e32', 'network_type': 'vxlan',
15:10:09 sean-k-mooney 'physical_network': None, 'segmentation_id': 1, 'network_id': 'ff960d9f-3b68-4b9b-8d69-78fe6441f27b'}}] {{(pid=90353) _bind_port_level /opt/stack/neutron/neutron/plugins/ml2/managers.py:948}}
15:10:11 frickler ah. maybe it is side effect of the segments test that runs in parallel. breaking other random tests
15:10:32 sean-k-mooney no right after ^ where the mech driver is able to bind
15:10:41 sean-k-mooney we get a concurrent bind excption
15:11:31 sean-k-mooney then neutron retires
15:11:31 frickler https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/job-output.txt#22299
15:11:34 sean-k-mooney 7.556330 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: WARNING neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Concurrent port binding operations failed on port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:35 sean-k-mooney Jun 30 11:36:27.557557 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: INFO neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Attempt 2 to bind port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:47 frickler look at ^^ the passing segment test right after the failure
15:11:59 frickler I'm pretty sure this is related
15:12:14 sean-k-mooney unless you are using the same port in both tests i dont see how it would be
15:12:24 frickler the segment test locking the DB leading to retries in other actions
15:12:59 sean-k-mooney oh so you think this trace https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10722
15:13:05 sean-k-mooney is caused by the segment test
15:13:40 sean-k-mooney ORM session: SQL execution without transaction in progress, traceback
15:14:17 sean-k-mooney lets check the nova logs and see if there is a rety on our side
15:14:27 sean-k-mooney if not then its an internal neutron issue
15:14:53 stephenfin sean-k-mooney: I'm not sure if that's related to this issue or not
15:16:50 stephenfin actually no, maybe it is. It's an update_port call that's causing the issue
15:31:49 opendevreview Merged openstack/nova stable/xena: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833411
16:04:55 opendevreview Merged openstack/os-vif master: Check for hybrid plugging in OVS https://review.opendev.org/c/openstack/os-vif/+/839102
16:37:44 opendevreview Merged openstack/nova master: ignore deleted server groups in validation https://review.opendev.org/c/openstack/nova/+/847001
16:54:38 opendevreview Merged openstack/nova stable/wallaby: compute: Ensure updates to bdms during pre_live_migration are saved https://review.opendev.org/c/openstack/nova/+/843680
16:54:45 opendevreview Merged openstack/nova stable/wallaby: fup: Make connection_info returned by CinderFixture unique per attachment https://review.opendev.org/c/openstack/nova/+/844594
16:54:51 opendevreview Merged openstack/nova stable/wallaby: fup: Assert state of connection_info during LM rollback in func tests https://review.opendev.org/c/openstack/nova/+/844595
18:44:31 opendevreview Jay Faulkner proposed openstack/nova stable/victoria: [ironic] Minimize window for a resource provider to be lost https://review.opendev.org/c/openstack/nova/+/800873
#openstack-nova - 2022-07-01
00:33:21 opendevreview melanie witt proposed openstack/nova master: libvirt: Report ephemeral encryption traits based on imagebackend https://review.opendev.org/c/openstack/nova/+/826753
06:55:42 kashyap Has anyone ever tested live migration with vTPM?
07:47:07 opendevreview Balazs Gibizer proposed openstack/nova master: Revert "zuul: Temporarly put Centos9 Stream job non-voting" https://review.opendev.org/c/openstack/nova/+/848352
07:47:21 gibi bauzas: ^^ I think we can turn it back on
08:15:09 bauzas gibi: I have some concerns with the Centos 9S job I wanted to discuss on the last meeting
08:15:19 gibi we can keep the revert open til next week
08:15:27 bauzas thanks
08:15:36 gibi I just noticed neutron reverted the skip so I thought I propose the revert to not to forget it
08:15:55 bauzas gibi: I'll add it into the agenda
08:15:59 gibi ack
10:57:41 manuvakery1 Hi .. user-data is not getting applied for any instance which is rebuild using a snapshot taken from the same instance
10:58:06 manuvakery1 reported a bug in https://bugs.launchpad.net/nova/+bug/1980511
11:04:33 sean-k-mooney that looks like a vlaid bug your runing openstack train
11:05:10 sean-k-mooney although i wonder
11:05:46 sean-k-mooney manuvakery1: before you snapshot the instnace did you remove the file that cloud init creates to indicate that it not a first boot
11:06:22 sean-k-mooney if you did not do that then it may not have downloaded an updated copy form teh metadata api
11:06:43 sean-k-mooney manuvakery1: are you using config drive ro jsut the metadata api?
11:12:27 manuvakery1 I am using metadata api
11:13:35 manuvakery1 sean-k-mooney: i have not removed any file which cloud-init created
11:14:40 sean-k-mooney ok then this is likely not a nova bug
11:15:06 sean-k-mooney i dont think cloud init will redonwload the user data file after first boot
11:16:37 sean-k-mooney manuvakery1: https://cloudinit.readthedocs.io/en/latest/topics/cli.html#clean
11:17:22 manuvakery1 but when I tried to rebuild with another snapshot ( created from another server) i see the user-data is getting downloaded
11:17:58 sean-k-mooney the donwloading of the user data is not somethign that nova is invovled with other then servicing it via the metadata api
11:18:10 sean-k-mooney have you checked that the content in the metadata api is correct
11:18:56 sean-k-mooney curl http://169.254.169.254/openstack/latest/user_data
11:19:23 sean-k-mooney if that has the correct content form within the vm then this is not a nova bug
11:19:52 manuvakery1 ok.. I will check that
11:20:02 frickler it may be an per-instance thing, depending on whether the snapshot contains the same instance-id as the rebuilt server
11:20:31 sean-k-mooney ya that too
11:20:46 sean-k-mooney usign the snapshot on a differnt vm might work
11:20:52 sean-k-mooney but not the one it was taken form
11:21:41 sean-k-mooney the instance id is in http://169.254.169.254/2009-04-04/meta-data/instance-id

Earlier   Later