Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-06-30
14:21:54 sean-k-mooney SMM is used for some other security feature and ring -2 hypervior feature at run time
14:21:57 mfo sean-k-mooney, IIUIC indeed, it's orthogonal, but it's involved in the implementation of not allowing the OS to tamper w/ the SB things in memory, otherwise the OS could bypass things.
14:22:12 mfo the link to pbonzini's presentation/video about it is really clarifying.
14:22:42 sean-k-mooney yep so in generall i would expect the default uefi image to be build with SMM_ENABLE
14:22:52 sean-k-mooney and the secure boot one to have require
14:24:26 bauzas sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/847001 +W with comments
14:24:57 bauzas sean-k-mooney: yeah but I had a lot of other stuff to do before :)
14:25:04 bauzas like the vGPU bugs
14:25:12 bauzas and then reviews and bug triage
14:25:22 bauzas (and now, back to be a mentor :) )
14:25:35 bauzas eventually, will work on this feature next week
14:25:42 bauzas maybe tomorrow if I have time
14:26:06 sean-k-mooney no rush sicne i actully had time to do upstream stuff today i was just looking to see what was close to merging
14:26:43 sean-k-mooney and the open review priorties
14:28:25 sean-k-mooney stephenfin: are you around today? if so got time to look at this os-vif patch its pretty small https://review.opendev.org/c/openstack/os-vif/+/839102
14:28:39 stephenfin sean-k-mooney: sure, I'll look now
14:29:07 sean-k-mooney ta
14:34:37 stephenfin done
14:36:44 sean-k-mooney many thanks
14:37:56 opendevreview Merged openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545
14:56:28 stephenfin sean-k-mooney: Seen this before? https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-n-cpu.txt#10060
14:56:34 stephenfin nova.exception.InternalError: Unexpected vif_type=unbound
14:56:46 stephenfin It rings a bell, but I thought we'd fixed this years ago
14:57:09 sean-k-mooney we had older bugs related to unbound
14:57:22 sean-k-mooney that is the state when the host-id is not set on the port
14:57:52 sean-k-mooney i think i looked at this
14:58:21 sean-k-mooney No conversion for VIF type unbound yet {{(pid=97953) nova_to_osvif_vif /opt/stack/nova/nova/network/os_vif_util.py:530}}
14:58:37 sean-k-mooney is really just a side effect of the prot not being bound properly in neutron
15:00:35 sean-k-mooney ah right
15:00:37 sean-k-mooney https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10891
15:00:50 sean-k-mooney so this is an issue with slow neutron i think
15:00:52 opendevreview Merged openstack/nova stable/wallaby: fake: Ensure need_legacy_block_device_info returns False https://review.opendev.org/c/openstack/nova/+/843678
15:00:59 opendevreview Merged openstack/nova stable/wallaby: Add a regression test for bug 1939545 https://review.opendev.org/c/openstack/nova/+/843679
15:01:22 sean-k-mooney stephenfin: basically when i was looking at this i was assumeing this happened because we retried the port bidning because neutron was slow to responed
15:01:44 sean-k-mooney and that caused a concurnet bind attempt that left it in ubound or something like that
15:02:10 stephenfin Hmm, that sounds reasonable. We saw it bubble up in the OSC tests because the server create failed. That sounds like a likely root cause though
15:02:46 sean-k-mooney if that is what is happening i woudl expect to see the retry logged somewher or see two bind attempts in the nutron log
15:02:54 sean-k-mooney ill check if i can see that
15:03:28 sean-k-mooney we do see the port had just finished binding when we got the concurrent error
15:03:29 sean-k-mooney ] Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id
15:05:10 sean-k-mooney ya it looks like there are 3 attempts to bind the port
15:05:23 frickler stephenfin: I mentioned that yesterday, it also seemed related to neutron retrying binds
15:05:26 sean-k-mooney in the neutron side the last to of which had the concurrent bind excption
15:05:45 sean-k-mooney frickler: well its actully the neutornclient retrying the bind
15:06:02 sean-k-mooney frickler: that was entirely broken in nova until somewhat recently
15:06:11 melwitt bauzas: I wanted to get your thoughts on this proposed patch to change logic in the placement audit nova-manage command, since you worked on it https://review.opendev.org/c/openstack/nova/+/844418 it seems like there is a bug in the current logic but it's not clear to me what the logic should be
15:06:23 sean-k-mooney frickler: we fixed retires about a year or so ago
15:06:30 bauzas melwitt: okay, I'll look
15:06:37 melwitt thanks
15:06:41 frickler sean-k-mooney: iiuc there is an internal retry in neutron happening now
15:07:04 sean-k-mooney frickler: there is also likely one in the db decorator
15:07:20 sean-k-mooney we can see transaction error in the log
15:07:42 frickler is this with neutron-segment enabled? we have some issue with segment ID reuse in OSC
15:08:34 sean-k-mooney i think segments were enabled yes
15:08:45 frickler oh, that's the osc job, yes
15:09:28 sean-k-mooney its using vxlan however i think
15:09:39 sean-k-mooney rather then routed provider networks
15:10:07 sean-k-mooney Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id': 'cd5c5c6b-1027-4fc7-bbc7-b8204df12e32', 'network_type': 'vxlan',
15:10:09 sean-k-mooney 'physical_network': None, 'segmentation_id': 1, 'network_id': 'ff960d9f-3b68-4b9b-8d69-78fe6441f27b'}}] {{(pid=90353) _bind_port_level /opt/stack/neutron/neutron/plugins/ml2/managers.py:948}}
15:10:11 frickler ah. maybe it is side effect of the segments test that runs in parallel. breaking other random tests
15:10:32 sean-k-mooney no right after ^ where the mech driver is able to bind
15:10:41 sean-k-mooney we get a concurrent bind excption
15:11:31 frickler https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/job-output.txt#22299
15:11:31 sean-k-mooney then neutron retires
15:11:34 sean-k-mooney 7.556330 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: WARNING neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Concurrent port binding operations failed on port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:35 sean-k-mooney Jun 30 11:36:27.557557 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: INFO neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Attempt 2 to bind port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:47 frickler look at ^^ the passing segment test right after the failure
15:11:59 frickler I'm pretty sure this is related
15:12:14 sean-k-mooney unless you are using the same port in both tests i dont see how it would be
15:12:24 frickler the segment test locking the DB leading to retries in other actions
15:12:59 sean-k-mooney oh so you think this trace https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10722
15:13:05 sean-k-mooney is caused by the segment test
15:13:40 sean-k-mooney ORM session: SQL execution without transaction in progress, traceback
15:14:17 sean-k-mooney lets check the nova logs and see if there is a rety on our side
15:14:27 sean-k-mooney if not then its an internal neutron issue
15:14:53 stephenfin sean-k-mooney: I'm not sure if that's related to this issue or not
15:16:50 stephenfin actually no, maybe it is. It's an update_port call that's causing the issue
15:31:49 opendevreview Merged openstack/nova stable/xena: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833411
16:04:55 opendevreview Merged openstack/os-vif master: Check for hybrid plugging in OVS https://review.opendev.org/c/openstack/os-vif/+/839102
16:37:44 opendevreview Merged openstack/nova master: ignore deleted server groups in validation https://review.opendev.org/c/openstack/nova/+/847001
16:54:38 opendevreview Merged openstack/nova stable/wallaby: compute: Ensure updates to bdms during pre_live_migration are saved https://review.opendev.org/c/openstack/nova/+/843680
16:54:45 opendevreview Merged openstack/nova stable/wallaby: fup: Make connection_info returned by CinderFixture unique per attachment https://review.opendev.org/c/openstack/nova/+/844594
16:54:51 opendevreview Merged openstack/nova stable/wallaby: fup: Assert state of connection_info during LM rollback in func tests https://review.opendev.org/c/openstack/nova/+/844595
18:44:31 opendevreview Jay Faulkner proposed openstack/nova stable/victoria: [ironic] Minimize window for a resource provider to be lost https://review.opendev.org/c/openstack/nova/+/800873
#openstack-nova - 2022-07-01
00:33:21 opendevreview melanie witt proposed openstack/nova master: libvirt: Report ephemeral encryption traits based on imagebackend https://review.opendev.org/c/openstack/nova/+/826753
06:55:42 kashyap Has anyone ever tested live migration with vTPM?
07:47:07 opendevreview Balazs Gibizer proposed openstack/nova master: Revert "zuul: Temporarly put Centos9 Stream job non-voting" https://review.opendev.org/c/openstack/nova/+/848352
07:47:21 gibi bauzas: ^^ I think we can turn it back on
08:15:09 bauzas gibi: I have some concerns with the Centos 9S job I wanted to discuss on the last meeting
08:15:19 gibi we can keep the revert open til next week
08:15:27 bauzas thanks
08:15:36 gibi I just noticed neutron reverted the skip so I thought I propose the revert to not to forget it
08:15:55 bauzas gibi: I'll add it into the agenda
08:15:59 gibi ack
10:57:41 manuvakery1 Hi .. user-data is not getting applied for any instance which is rebuild using a snapshot taken from the same instance
10:58:06 manuvakery1 reported a bug in https://bugs.launchpad.net/nova/+bug/1980511
11:04:33 sean-k-mooney that looks like a vlaid bug your runing openstack train
11:05:10 sean-k-mooney although i wonder
11:05:46 sean-k-mooney manuvakery1: before you snapshot the instnace did you remove the file that cloud init creates to indicate that it not a first boot
11:06:22 sean-k-mooney if you did not do that then it may not have downloaded an updated copy form teh metadata api

Earlier   Later