Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-06-30
14:04:35 sean-k-mooney but yes sercure boot need uefi
14:04:35 mfo sean-k-mooney, yup. this is currently how it is, but the for-loop doesn't have a `break` in case it finds one, so the last file is used.
14:05:06 mfo i could just add a break in there, but this would change things / file end up chosen in the general case, i guess.
14:05:11 sean-k-mooney so it wont work with hw_machine_type=pc`
14:05:42 mfo and for older stable branches, well, i didnt want to risk such changes. (i'm not too familiar w/ openstack yet, actually).
14:07:14 sean-k-mooney i think we aleay want to break on the first file we find that exsits as long as we make the secure boot one last if you did not ask for secure boot
14:07:47 sean-k-mooney we are only going to boot the vm once anyway so there is no point in continuing to check
14:08:09 sean-k-mooney but i dont have your patch open currently so that depnds on how you wrote the loop
14:09:30 opendevreview ribaudr proposed openstack/nova master: Allow unshelve to a specific host (Compute API part) https://review.opendev.org/c/openstack/nova/+/831507
14:09:31 opendevreview ribaudr proposed openstack/nova master: Allow unshelve to a specific host (REST API part) https://review.opendev.org/c/openstack/nova/+/845897
14:10:07 mfo right, i wondered why the (existing) loop didn't have a 'break' statement in the first place when it was introduced, but it was long ago, and didn't seem like a thing to change for old stable branches. also, in this version one can't ask for secure boot yet (this is victoria/ussuri, and secureboot came in wallaby), it just happened that a patch adding the OVMF paths included a .secboot.fd file, before proper SB support came in.
14:11:31 sean-k-mooney ack
14:12:30 sean-k-mooney the ovmf frimware images can be build so that secure boot is avaiable but not required
14:12:46 sean-k-mooney so i think the old logic was intended to allow it to be used if it was supproted
14:12:58 sean-k-mooney by the ovmf image by manually configuring it via the boot menu
14:13:04 sean-k-mooney but that was not really supported by nova
14:13:10 mfo sean-k-mooney, ah, i see.
14:14:19 mfo i think that works for the SB feature alone, but i think the SMM feature support once it's built in then it isn't opt-in, right? than that requires support in the emulator (ie, qemu's q35).
14:14:37 mfo s/than/then/
14:14:52 sean-k-mooney i think that is correct but dont know all the details
14:15:02 sean-k-mooney SMM does require qemu to be configured to enable it
14:15:23 sean-k-mooney but i dont know if you need that enabled if its complied in to the ovmf image
14:15:41 mfo yes, there are many details in this. i added some docs/links to the patch for the research i had done, but it's indeed at the lower-level details.
14:15:41 sean-k-mooney its all a bit of a mess
14:17:11 bauzas sean-k-mooney! sorry, haven't yet done the implementation for the keypair deprecation
14:19:14 sean-k-mooney bauzas: ack no worries i was just expecting that to be relitivly small so if it was off your backlog you woudl have more time to reivew
14:19:19 mfo sean-k-mooney, if you're ever curious, i doc'ed the research details of OVMF/SB/SMM/QEMU in bug 1960758 comment #6, the takeaway is, you can build OVMF with SMM_ENABLE, but then it's not really _secure_ Secure Boot; for that you need SMM_REQUIRE, and then platform support is not optional, it's required as well.
14:19:52 sean-k-mooney bauzas: i was just trying to see if we could reduce context switching by merging some small easy wins before m2
14:20:04 sean-k-mooney to have less to context switch between coming up to FF
14:20:48 sean-k-mooney mfo: SMM is system management mode support which is actuly unrealted to secure boot
14:21:54 sean-k-mooney SMM is used for some other security feature and ring -2 hypervior feature at run time
14:21:57 mfo sean-k-mooney, IIUIC indeed, it's orthogonal, but it's involved in the implementation of not allowing the OS to tamper w/ the SB things in memory, otherwise the OS could bypass things.
14:22:12 mfo the link to pbonzini's presentation/video about it is really clarifying.
14:22:42 sean-k-mooney yep so in generall i would expect the default uefi image to be build with SMM_ENABLE
14:22:52 sean-k-mooney and the secure boot one to have require
14:24:26 bauzas sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/847001 +W with comments
14:24:57 bauzas sean-k-mooney: yeah but I had a lot of other stuff to do before :)
14:25:04 bauzas like the vGPU bugs
14:25:12 bauzas and then reviews and bug triage
14:25:22 bauzas (and now, back to be a mentor :) )
14:25:35 bauzas eventually, will work on this feature next week
14:25:42 bauzas maybe tomorrow if I have time
14:26:06 sean-k-mooney no rush sicne i actully had time to do upstream stuff today i was just looking to see what was close to merging
14:26:43 sean-k-mooney and the open review priorties
14:28:25 sean-k-mooney stephenfin: are you around today? if so got time to look at this os-vif patch its pretty small https://review.opendev.org/c/openstack/os-vif/+/839102
14:28:39 stephenfin sean-k-mooney: sure, I'll look now
14:29:07 sean-k-mooney ta
14:34:37 stephenfin done
14:36:44 sean-k-mooney many thanks
14:37:56 opendevreview Merged openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545
14:56:28 stephenfin sean-k-mooney: Seen this before? https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-n-cpu.txt#10060
14:56:34 stephenfin nova.exception.InternalError: Unexpected vif_type=unbound
14:56:46 stephenfin It rings a bell, but I thought we'd fixed this years ago
14:57:09 sean-k-mooney we had older bugs related to unbound
14:57:22 sean-k-mooney that is the state when the host-id is not set on the port
14:57:52 sean-k-mooney i think i looked at this
14:58:21 sean-k-mooney No conversion for VIF type unbound yet {{(pid=97953) nova_to_osvif_vif /opt/stack/nova/nova/network/os_vif_util.py:530}}
14:58:37 sean-k-mooney is really just a side effect of the prot not being bound properly in neutron
15:00:35 sean-k-mooney ah right
15:00:37 sean-k-mooney https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10891
15:00:50 sean-k-mooney so this is an issue with slow neutron i think
15:00:52 opendevreview Merged openstack/nova stable/wallaby: fake: Ensure need_legacy_block_device_info returns False https://review.opendev.org/c/openstack/nova/+/843678
15:00:59 opendevreview Merged openstack/nova stable/wallaby: Add a regression test for bug 1939545 https://review.opendev.org/c/openstack/nova/+/843679
15:01:22 sean-k-mooney stephenfin: basically when i was looking at this i was assumeing this happened because we retried the port bidning because neutron was slow to responed
15:01:44 sean-k-mooney and that caused a concurnet bind attempt that left it in ubound or something like that
15:02:10 stephenfin Hmm, that sounds reasonable. We saw it bubble up in the OSC tests because the server create failed. That sounds like a likely root cause though
15:02:46 sean-k-mooney if that is what is happening i woudl expect to see the retry logged somewher or see two bind attempts in the nutron log
15:02:54 sean-k-mooney ill check if i can see that
15:03:28 sean-k-mooney we do see the port had just finished binding when we got the concurrent error
15:03:29 sean-k-mooney ] Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id
15:05:10 sean-k-mooney ya it looks like there are 3 attempts to bind the port
15:05:23 frickler stephenfin: I mentioned that yesterday, it also seemed related to neutron retrying binds
15:05:26 sean-k-mooney in the neutron side the last to of which had the concurrent bind excption
15:05:45 sean-k-mooney frickler: well its actully the neutornclient retrying the bind
15:06:02 sean-k-mooney frickler: that was entirely broken in nova until somewhat recently
15:06:11 melwitt bauzas: I wanted to get your thoughts on this proposed patch to change logic in the placement audit nova-manage command, since you worked on it https://review.opendev.org/c/openstack/nova/+/844418 it seems like there is a bug in the current logic but it's not clear to me what the logic should be
15:06:23 sean-k-mooney frickler: we fixed retires about a year or so ago
15:06:30 bauzas melwitt: okay, I'll look
15:06:37 melwitt thanks
15:06:41 frickler sean-k-mooney: iiuc there is an internal retry in neutron happening now
15:07:04 sean-k-mooney frickler: there is also likely one in the db decorator
15:07:20 sean-k-mooney we can see transaction error in the log
15:07:42 frickler is this with neutron-segment enabled? we have some issue with segment ID reuse in OSC
15:08:34 sean-k-mooney i think segments were enabled yes
15:08:45 frickler oh, that's the osc job, yes
15:09:28 sean-k-mooney its using vxlan however i think
15:09:39 sean-k-mooney rather then routed provider networks
15:10:07 sean-k-mooney Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id': 'cd5c5c6b-1027-4fc7-bbc7-b8204df12e32', 'network_type': 'vxlan',
15:10:09 sean-k-mooney 'physical_network': None, 'segmentation_id': 1, 'network_id': 'ff960d9f-3b68-4b9b-8d69-78fe6441f27b'}}] {{(pid=90353) _bind_port_level /opt/stack/neutron/neutron/plugins/ml2/managers.py:948}}
15:10:11 frickler ah. maybe it is side effect of the segments test that runs in parallel. breaking other random tests
15:10:32 sean-k-mooney no right after ^ where the mech driver is able to bind
15:10:41 sean-k-mooney we get a concurrent bind excption
15:11:31 frickler https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/job-output.txt#22299
15:11:31 sean-k-mooney then neutron retires
15:11:34 sean-k-mooney 7.556330 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: WARNING neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Concurrent port binding operations failed on port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:35 sean-k-mooney Jun 30 11:36:27.557557 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: INFO neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Attempt 2 to bind port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:47 frickler look at ^^ the passing segment test right after the failure
15:11:59 frickler I'm pretty sure this is related
15:12:14 sean-k-mooney unless you are using the same port in both tests i dont see how it would be
15:12:24 frickler the segment test locking the DB leading to retries in other actions

Earlier   Later