Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-06-30
14:08:09 sean-k-mooney but i dont have your patch open currently so that depnds on how you wrote the loop
14:09:30 opendevreview ribaudr proposed openstack/nova master: Allow unshelve to a specific host (Compute API part) https://review.opendev.org/c/openstack/nova/+/831507
14:09:31 opendevreview ribaudr proposed openstack/nova master: Allow unshelve to a specific host (REST API part) https://review.opendev.org/c/openstack/nova/+/845897
14:10:07 mfo right, i wondered why the (existing) loop didn't have a 'break' statement in the first place when it was introduced, but it was long ago, and didn't seem like a thing to change for old stable branches. also, in this version one can't ask for secure boot yet (this is victoria/ussuri, and secureboot came in wallaby), it just happened that a patch adding the OVMF paths included a .secboot.fd file, before proper SB support came in.
14:11:31 sean-k-mooney ack
14:12:30 sean-k-mooney the ovmf frimware images can be build so that secure boot is avaiable but not required
14:12:46 sean-k-mooney so i think the old logic was intended to allow it to be used if it was supproted
14:12:58 sean-k-mooney by the ovmf image by manually configuring it via the boot menu
14:13:04 sean-k-mooney but that was not really supported by nova
14:13:10 mfo sean-k-mooney, ah, i see.
14:14:19 mfo i think that works for the SB feature alone, but i think the SMM feature support once it's built in then it isn't opt-in, right? than that requires support in the emulator (ie, qemu's q35).
14:14:37 mfo s/than/then/
14:14:52 sean-k-mooney i think that is correct but dont know all the details
14:15:02 sean-k-mooney SMM does require qemu to be configured to enable it
14:15:23 sean-k-mooney but i dont know if you need that enabled if its complied in to the ovmf image
14:15:41 sean-k-mooney its all a bit of a mess
14:15:41 mfo yes, there are many details in this. i added some docs/links to the patch for the research i had done, but it's indeed at the lower-level details.
14:17:11 bauzas sean-k-mooney! sorry, haven't yet done the implementation for the keypair deprecation
14:19:14 sean-k-mooney bauzas: ack no worries i was just expecting that to be relitivly small so if it was off your backlog you woudl have more time to reivew
14:19:19 mfo sean-k-mooney, if you're ever curious, i doc'ed the research details of OVMF/SB/SMM/QEMU in bug 1960758 comment #6, the takeaway is, you can build OVMF with SMM_ENABLE, but then it's not really _secure_ Secure Boot; for that you need SMM_REQUIRE, and then platform support is not optional, it's required as well.
14:19:52 sean-k-mooney bauzas: i was just trying to see if we could reduce context switching by merging some small easy wins before m2
14:20:04 sean-k-mooney to have less to context switch between coming up to FF
14:20:48 sean-k-mooney mfo: SMM is system management mode support which is actuly unrealted to secure boot
14:21:54 sean-k-mooney SMM is used for some other security feature and ring -2 hypervior feature at run time
14:21:57 mfo sean-k-mooney, IIUIC indeed, it's orthogonal, but it's involved in the implementation of not allowing the OS to tamper w/ the SB things in memory, otherwise the OS could bypass things.
14:22:12 mfo the link to pbonzini's presentation/video about it is really clarifying.
14:22:42 sean-k-mooney yep so in generall i would expect the default uefi image to be build with SMM_ENABLE
14:22:52 sean-k-mooney and the secure boot one to have require
14:24:26 bauzas sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/847001 +W with comments
14:24:57 bauzas sean-k-mooney: yeah but I had a lot of other stuff to do before :)
14:25:04 bauzas like the vGPU bugs
14:25:12 bauzas and then reviews and bug triage
14:25:22 bauzas (and now, back to be a mentor :) )
14:25:35 bauzas eventually, will work on this feature next week
14:25:42 bauzas maybe tomorrow if I have time
14:26:06 sean-k-mooney no rush sicne i actully had time to do upstream stuff today i was just looking to see what was close to merging
14:26:43 sean-k-mooney and the open review priorties
14:28:25 sean-k-mooney stephenfin: are you around today? if so got time to look at this os-vif patch its pretty small https://review.opendev.org/c/openstack/os-vif/+/839102
14:28:39 stephenfin sean-k-mooney: sure, I'll look now
14:29:07 sean-k-mooney ta
14:34:37 stephenfin done
14:36:44 sean-k-mooney many thanks
14:37:56 opendevreview Merged openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545
14:56:28 stephenfin sean-k-mooney: Seen this before? https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-n-cpu.txt#10060
14:56:34 stephenfin nova.exception.InternalError: Unexpected vif_type=unbound
14:56:46 stephenfin It rings a bell, but I thought we'd fixed this years ago
14:57:09 sean-k-mooney we had older bugs related to unbound
14:57:22 sean-k-mooney that is the state when the host-id is not set on the port
14:57:52 sean-k-mooney i think i looked at this
14:58:21 sean-k-mooney No conversion for VIF type unbound yet {{(pid=97953) nova_to_osvif_vif /opt/stack/nova/nova/network/os_vif_util.py:530}}
14:58:37 sean-k-mooney is really just a side effect of the prot not being bound properly in neutron
15:00:35 sean-k-mooney ah right
15:00:37 sean-k-mooney https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10891
15:00:50 sean-k-mooney so this is an issue with slow neutron i think
15:00:52 opendevreview Merged openstack/nova stable/wallaby: fake: Ensure need_legacy_block_device_info returns False https://review.opendev.org/c/openstack/nova/+/843678
15:00:59 opendevreview Merged openstack/nova stable/wallaby: Add a regression test for bug 1939545 https://review.opendev.org/c/openstack/nova/+/843679
15:01:22 sean-k-mooney stephenfin: basically when i was looking at this i was assumeing this happened because we retried the port bidning because neutron was slow to responed
15:01:44 sean-k-mooney and that caused a concurnet bind attempt that left it in ubound or something like that
15:02:10 stephenfin Hmm, that sounds reasonable. We saw it bubble up in the OSC tests because the server create failed. That sounds like a likely root cause though
15:02:46 sean-k-mooney if that is what is happening i woudl expect to see the retry logged somewher or see two bind attempts in the nutron log
15:02:54 sean-k-mooney ill check if i can see that
15:03:28 sean-k-mooney we do see the port had just finished binding when we got the concurrent error
15:03:29 sean-k-mooney ] Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id
15:05:10 sean-k-mooney ya it looks like there are 3 attempts to bind the port
15:05:23 frickler stephenfin: I mentioned that yesterday, it also seemed related to neutron retrying binds
15:05:26 sean-k-mooney in the neutron side the last to of which had the concurrent bind excption
15:05:45 sean-k-mooney frickler: well its actully the neutornclient retrying the bind
15:06:02 sean-k-mooney frickler: that was entirely broken in nova until somewhat recently
15:06:11 melwitt bauzas: I wanted to get your thoughts on this proposed patch to change logic in the placement audit nova-manage command, since you worked on it https://review.opendev.org/c/openstack/nova/+/844418 it seems like there is a bug in the current logic but it's not clear to me what the logic should be
15:06:23 sean-k-mooney frickler: we fixed retires about a year or so ago
15:06:30 bauzas melwitt: okay, I'll look
15:06:37 melwitt thanks
15:06:41 frickler sean-k-mooney: iiuc there is an internal retry in neutron happening now
15:07:04 sean-k-mooney frickler: there is also likely one in the db decorator
15:07:20 sean-k-mooney we can see transaction error in the log
15:07:42 frickler is this with neutron-segment enabled? we have some issue with segment ID reuse in OSC
15:08:34 sean-k-mooney i think segments were enabled yes
15:08:45 frickler oh, that's the osc job, yes
15:09:28 sean-k-mooney its using vxlan however i think
15:09:39 sean-k-mooney rather then routed provider networks
15:10:07 sean-k-mooney Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id': 'cd5c5c6b-1027-4fc7-bbc7-b8204df12e32', 'network_type': 'vxlan',
15:10:09 sean-k-mooney 'physical_network': None, 'segmentation_id': 1, 'network_id': 'ff960d9f-3b68-4b9b-8d69-78fe6441f27b'}}] {{(pid=90353) _bind_port_level /opt/stack/neutron/neutron/plugins/ml2/managers.py:948}}
15:10:11 frickler ah. maybe it is side effect of the segments test that runs in parallel. breaking other random tests
15:10:32 sean-k-mooney no right after ^ where the mech driver is able to bind
15:10:41 sean-k-mooney we get a concurrent bind excption
15:11:31 sean-k-mooney then neutron retires
15:11:31 frickler https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/job-output.txt#22299
15:11:34 sean-k-mooney 7.556330 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: WARNING neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Concurrent port binding operations failed on port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:35 sean-k-mooney Jun 30 11:36:27.557557 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: INFO neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Attempt 2 to bind port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:47 frickler look at ^^ the passing segment test right after the failure
15:11:59 frickler I'm pretty sure this is related
15:12:14 sean-k-mooney unless you are using the same port in both tests i dont see how it would be
15:12:24 frickler the segment test locking the DB leading to retries in other actions
15:12:59 sean-k-mooney oh so you think this trace https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10722
15:13:05 sean-k-mooney is caused by the segment test
15:13:40 sean-k-mooney ORM session: SQL execution without transaction in progress, traceback
15:14:17 sean-k-mooney lets check the nova logs and see if there is a rety on our side
15:14:27 sean-k-mooney if not then its an internal neutron issue
15:14:53 stephenfin sean-k-mooney: I'm not sure if that's related to this issue or not
15:16:50 stephenfin actually no, maybe it is. It's an update_port call that's causing the issue

Earlier   Later