| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-30 | |||
| 14:02:30 | mfo | so we should *try* other firmware/loader files, or use .secboot.fd anyway if its' the only one that exists (not to break/change from what is done in this old/stable branch). | |
| 14:02:41 | mfo | This was a suggestion from sean-k-mooney, and also to have unit tests. | |
| 14:02:47 | mfo | Your suggestions were 1) to convert the list of firmware/loader paths to constants (so to intentionally "break" any downstream stable consumers that possibly patched that, so they'd be aware of the change and review their version), and 2) change the patch approach to not even iterate the list of firmware/loader files with .secboot.fd if we're on PC machine type. (I just combined it with, "ok, use it if it's the only one" per Sean's | |
| 14:02:47 | mfo | suggestion too). | |
| 14:03:51 | sean-k-mooney | i think i basicaly said sort so that its last in the list and if its the only one then use it and maybe it will work | |
| 14:03:51 | mfo | I guess that's what this is about and where we are. :) | |
| 14:03:58 | sean-k-mooney | rahter then deleteing it form the list | |
| 14:04:35 | mfo | sean-k-mooney, yup. this is currently how it is, but the for-loop doesn't have a `break` in case it finds one, so the last file is used. | |
| 14:04:35 | sean-k-mooney | but yes sercure boot need uefi | |
| 14:05:06 | mfo | i could just add a break in there, but this would change things / file end up chosen in the general case, i guess. | |
| 14:05:11 | sean-k-mooney | so it wont work with hw_machine_type=pc` | |
| 14:05:42 | mfo | and for older stable branches, well, i didnt want to risk such changes. (i'm not too familiar w/ openstack yet, actually). | |
| 14:07:14 | sean-k-mooney | i think we aleay want to break on the first file we find that exsits as long as we make the secure boot one last if you did not ask for secure boot | |
| 14:07:47 | sean-k-mooney | we are only going to boot the vm once anyway so there is no point in continuing to check | |
| 14:08:09 | sean-k-mooney | but i dont have your patch open currently so that depnds on how you wrote the loop | |
| 14:09:30 | opendevreview | ribaudr proposed openstack/nova master: Allow unshelve to a specific host (Compute API part) https://review.opendev.org/c/openstack/nova/+/831507 | |
| 14:09:31 | opendevreview | ribaudr proposed openstack/nova master: Allow unshelve to a specific host (REST API part) https://review.opendev.org/c/openstack/nova/+/845897 | |
| 14:10:07 | mfo | right, i wondered why the (existing) loop didn't have a 'break' statement in the first place when it was introduced, but it was long ago, and didn't seem like a thing to change for old stable branches. also, in this version one can't ask for secure boot yet (this is victoria/ussuri, and secureboot came in wallaby), it just happened that a patch adding the OVMF paths included a .secboot.fd file, before proper SB support came in. | |
| 14:11:31 | sean-k-mooney | ack | |
| 14:12:30 | sean-k-mooney | the ovmf frimware images can be build so that secure boot is avaiable but not required | |
| 14:12:46 | sean-k-mooney | so i think the old logic was intended to allow it to be used if it was supproted | |
| 14:12:58 | sean-k-mooney | by the ovmf image by manually configuring it via the boot menu | |
| 14:13:04 | sean-k-mooney | but that was not really supported by nova | |
| 14:13:10 | mfo | sean-k-mooney, ah, i see. | |
| 14:14:19 | mfo | i think that works for the SB feature alone, but i think the SMM feature support once it's built in then it isn't opt-in, right? than that requires support in the emulator (ie, qemu's q35). | |
| 14:14:37 | mfo | s/than/then/ | |
| 14:14:52 | sean-k-mooney | i think that is correct but dont know all the details | |
| 14:15:02 | sean-k-mooney | SMM does require qemu to be configured to enable it | |
| 14:15:23 | sean-k-mooney | but i dont know if you need that enabled if its complied in to the ovmf image | |
| 14:15:41 | sean-k-mooney | its all a bit of a mess | |
| 14:15:41 | mfo | yes, there are many details in this. i added some docs/links to the patch for the research i had done, but it's indeed at the lower-level details. | |
| 14:17:11 | bauzas | sean-k-mooney! sorry, haven't yet done the implementation for the keypair deprecation | |
| 14:19:14 | sean-k-mooney | bauzas: ack no worries i was just expecting that to be relitivly small so if it was off your backlog you woudl have more time to reivew | |
| 14:19:19 | mfo | sean-k-mooney, if you're ever curious, i doc'ed the research details of OVMF/SB/SMM/QEMU in bug 1960758 comment #6, the takeaway is, you can build OVMF with SMM_ENABLE, but then it's not really _secure_ Secure Boot; for that you need SMM_REQUIRE, and then platform support is not optional, it's required as well. | |
| 14:19:52 | sean-k-mooney | bauzas: i was just trying to see if we could reduce context switching by merging some small easy wins before m2 | |
| 14:20:04 | sean-k-mooney | to have less to context switch between coming up to FF | |
| 14:20:48 | sean-k-mooney | mfo: SMM is system management mode support which is actuly unrealted to secure boot | |
| 14:21:54 | sean-k-mooney | SMM is used for some other security feature and ring -2 hypervior feature at run time | |
| 14:21:57 | mfo | sean-k-mooney, IIUIC indeed, it's orthogonal, but it's involved in the implementation of not allowing the OS to tamper w/ the SB things in memory, otherwise the OS could bypass things. | |
| 14:22:12 | mfo | the link to pbonzini's presentation/video about it is really clarifying. | |
| 14:22:42 | sean-k-mooney | yep so in generall i would expect the default uefi image to be build with SMM_ENABLE | |
| 14:22:52 | sean-k-mooney | and the secure boot one to have require | |
| 14:24:26 | bauzas | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/847001 +W with comments | |
| 14:24:57 | bauzas | sean-k-mooney: yeah but I had a lot of other stuff to do before :) | |
| 14:25:04 | bauzas | like the vGPU bugs | |
| 14:25:12 | bauzas | and then reviews and bug triage | |
| 14:25:22 | bauzas | (and now, back to be a mentor :) ) | |
| 14:25:35 | bauzas | eventually, will work on this feature next week | |
| 14:25:42 | bauzas | maybe tomorrow if I have time | |
| 14:26:06 | sean-k-mooney | no rush sicne i actully had time to do upstream stuff today i was just looking to see what was close to merging | |
| 14:26:43 | sean-k-mooney | and the open review priorties | |
| 14:28:25 | sean-k-mooney | stephenfin: are you around today? if so got time to look at this os-vif patch its pretty small https://review.opendev.org/c/openstack/os-vif/+/839102 | |
| 14:28:39 | stephenfin | sean-k-mooney: sure, I'll look now | |
| 14:29:07 | sean-k-mooney | ta | |
| 14:34:37 | stephenfin | done | |
| 14:36:44 | sean-k-mooney | many thanks | |
| 14:37:56 | opendevreview | Merged openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545 | |
| 14:56:28 | stephenfin | sean-k-mooney: Seen this before? https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-n-cpu.txt#10060 | |
| 14:56:34 | stephenfin | nova.exception.InternalError: Unexpected vif_type=unbound | |
| 14:56:46 | stephenfin | It rings a bell, but I thought we'd fixed this years ago | |
| 14:57:09 | sean-k-mooney | we had older bugs related to unbound | |
| 14:57:22 | sean-k-mooney | that is the state when the host-id is not set on the port | |
| 14:57:52 | sean-k-mooney | i think i looked at this | |
| 14:58:21 | sean-k-mooney | No conversion for VIF type unbound yet {{(pid=97953) nova_to_osvif_vif /opt/stack/nova/nova/network/os_vif_util.py:530}} | |
| 14:58:37 | sean-k-mooney | is really just a side effect of the prot not being bound properly in neutron | |
| 15:00:35 | sean-k-mooney | ah right | |
| 15:00:37 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10891 | |
| 15:00:50 | sean-k-mooney | so this is an issue with slow neutron i think | |
| 15:00:52 | opendevreview | Merged openstack/nova stable/wallaby: fake: Ensure need_legacy_block_device_info returns False https://review.opendev.org/c/openstack/nova/+/843678 | |
| 15:00:59 | opendevreview | Merged openstack/nova stable/wallaby: Add a regression test for bug 1939545 https://review.opendev.org/c/openstack/nova/+/843679 | |
| 15:01:22 | sean-k-mooney | stephenfin: basically when i was looking at this i was assumeing this happened because we retried the port bidning because neutron was slow to responed | |
| 15:01:44 | sean-k-mooney | and that caused a concurnet bind attempt that left it in ubound or something like that | |
| 15:02:10 | stephenfin | Hmm, that sounds reasonable. We saw it bubble up in the OSC tests because the server create failed. That sounds like a likely root cause though | |
| 15:02:46 | sean-k-mooney | if that is what is happening i woudl expect to see the retry logged somewher or see two bind attempts in the nutron log | |
| 15:02:54 | sean-k-mooney | ill check if i can see that | |
| 15:03:28 | sean-k-mooney | we do see the port had just finished binding when we got the concurrent error | |
| 15:03:29 | sean-k-mooney | ] Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id | |
| 15:05:10 | sean-k-mooney | ya it looks like there are 3 attempts to bind the port | |
| 15:05:23 | frickler | stephenfin: I mentioned that yesterday, it also seemed related to neutron retrying binds | |
| 15:05:26 | sean-k-mooney | in the neutron side the last to of which had the concurrent bind excption | |
| 15:05:45 | sean-k-mooney | frickler: well its actully the neutornclient retrying the bind | |
| 15:06:02 | sean-k-mooney | frickler: that was entirely broken in nova until somewhat recently | |
| 15:06:11 | melwitt | bauzas: I wanted to get your thoughts on this proposed patch to change logic in the placement audit nova-manage command, since you worked on it https://review.opendev.org/c/openstack/nova/+/844418 it seems like there is a bug in the current logic but it's not clear to me what the logic should be | |
| 15:06:23 | sean-k-mooney | frickler: we fixed retires about a year or so ago | |
| 15:06:30 | bauzas | melwitt: okay, I'll look | |
| 15:06:37 | melwitt | thanks | |
| 15:06:41 | frickler | sean-k-mooney: iiuc there is an internal retry in neutron happening now | |
| 15:07:04 | sean-k-mooney | frickler: there is also likely one in the db decorator | |
| 15:07:20 | sean-k-mooney | we can see transaction error in the log | |
| 15:07:42 | frickler | is this with neutron-segment enabled? we have some issue with segment ID reuse in OSC | |
| 15:08:34 | sean-k-mooney | i think segments were enabled yes | |
| 15:08:45 | frickler | oh, that's the osc job, yes | |
| 15:09:28 | sean-k-mooney | its using vxlan however i think | |
| 15:09:39 | sean-k-mooney | rather then routed provider networks | |
| 15:10:07 | sean-k-mooney | Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id': 'cd5c5c6b-1027-4fc7-bbc7-b8204df12e32', 'network_type': 'vxlan', | |
| 15:10:09 | sean-k-mooney | 'physical_network': None, 'segmentation_id': 1, 'network_id': 'ff960d9f-3b68-4b9b-8d69-78fe6441f27b'}}] {{(pid=90353) _bind_port_level /opt/stack/neutron/neutron/plugins/ml2/managers.py:948}} | |
| 15:10:11 | frickler | ah. maybe it is side effect of the segments test that runs in parallel. breaking other random tests | |
| 15:10:32 | sean-k-mooney | no right after ^ where the mech driver is able to bind | |
| 15:10:41 | sean-k-mooney | we get a concurrent bind excption | |
| 15:11:31 | sean-k-mooney | then neutron retires | |