| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-23 | |||
| 13:38:13 | kashyap | Hmm, I'm trying to minimize the "impact surface" here. | |
| 13:38:47 | gibi | I guess we are back to the drawing board then | |
| 13:38:57 | kashyap | gibi: The first check was wrong anyway: so that we can remove w/o doubt. | |
| 13:39:10 | sean-k-mooney | no not in all cases | |
| 13:39:21 | sean-k-mooney | it was only wrogn if you enabled a modle that is not compatible with the cpu | |
| 13:39:21 | gibi | the two check uses a different cpu.model for comparision | |
| 13:39:39 | sean-k-mooney | gibi: right and the second check is not looping over the lsit fo cpu_modles | |
| 13:40:22 | sean-k-mooney | so we need to 1 loop over them using self._get_cpu_model_mapping(model) and 2 apply all cpu flags before doing the compare | |
| 13:41:17 | sean-k-mooney | 3 the comapre should not be condtional on extra flags and should happen even when empty | |
| 13:41:53 | kashyap | Hm, that could make whole code flow to be changed and a lot more surface impact :-( - I wonder if we try it on _top_ of the workaround? | |
| 13:41:57 | gibi | basically to see that each configred model with all the configured flags applied are still compatible with the hypervisor | |
| 13:42:39 | kashyap | gibi: Yeah, that's what libvirt will be doing under the hood "by default". | |
| 13:43:11 | gibi | kashyap: I don't think so. I guess libvirt will do the check during boot with the selected model + extra flags | |
| 13:43:24 | gibi | not each model + flags at startup | |
| 13:43:30 | kashyap | (I'm just thinking if we can "do this additional improvement" on _top_ of the current workaround + API replacement patch?) | |
| 13:43:56 | gibi | I guess if we want the workaround to land independently then we need to add back the old check and make both check optional with the WA flag. | |
| 13:44:12 | gibi | sean-k-mooney: ^^ what do you think? | |
| 13:44:39 | kashyap | Hmm, the first check is wrong unilaterally - sean-k-mooney agrees that too, IIUC | |
| 13:44:56 | kashyap | (As based on his suggestion is the current revision) | |
| 13:45:42 | kashyap | (And thanks for your patience so far, folks!) | |
| 13:45:55 | gibi | then I'm confused, as we discussed above that the second check alone is not enough. | |
| 13:46:07 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/870794/8/nova/virt/libvirt/driver.py#992 | |
| 13:46:14 | sean-k-mooney | kashyap: actully i dont | |
| 13:46:17 | gibi | as it does nothing when no extra flags I configured | |
| 13:46:30 | sean-k-mooney | kashyap: i said its condtionally wrong if and only if you use extra_flags | |
| 13:47:03 | sean-k-mooney | anyway i posted what i think is the correct code in the comment above ^ | |
| 13:48:16 | kashyap | sean-k-mooney: Well, I'm fine if you've refined your view _now_. But a few days ago you did write here: | |
| 13:48:19 | kashyap | (quote) | |
| 13:48:22 | kashyap | < sean-k-mooney> but if you want to future proof then sure you can put the remain check under a workaroudn | |
| 13:48:25 | kashyap | [...] | |
| 13:48:27 | kashyap | < sean-k-mooney> or just delete the first check | |
| 13:48:30 | kashyap | (/quote) | |
| 13:49:11 | kashyap | (Doing "nothing when no extra flags" are configured is still fine - as it was already explicitly tested by a downstream bug reporter in a real env) | |
| 13:49:19 | sean-k-mooney | yes i did not express my concern at that point with the looping as i tought i has already disucsst that with you | |
| 13:49:30 | kashyap | Okay. | |
| 13:49:59 | sean-k-mooney | kashyap: its not fine because its an upstream regression | |
| 13:50:00 | kashyap | gibi: sean-k-mooney: Also it was tested with the workaround as it stands: https://bugzilla.redhat.com/show_bug.cgi?id=2138381#c45 | |
| 13:50:12 | kashyap | (Unforunately, it's a private comment, so only RHTers can see it) | |
| 13:51:12 | kashyap | (Hmm, perhaps ignore that test, I think he tested it with the first revision) | |
| 13:51:25 | gibi | kashyap: you will only see the changed behaivor if the test is configuring incompatible model without extra flags. | |
| 13:51:50 | kashyap | gibi: True | |
| 13:51:55 | gibi | in the baseline that will fail the startup of nova-compute in the new code it will let the nova-compute start | |
| 13:52:08 | gibi | but fail all instance boot instead | |
| 13:52:15 | kashyap | I'm trying to find a reasonable compromise here: so shall I put both the blocks back into the code and add the workaround? | |
| 13:52:24 | kashyap | (s/blocks/calls/) | |
| 13:53:08 | kashyap | gibi: Right, failing at instance boot is still "something" - as it still provides a libvirt error. But I know we've already discussed and that's not 'acceptable' | |
| 13:53:50 | sean-k-mooney | i condier the bug you are introduceing by removign all cpu checks in the case that cpu flags is empty to be more serios then the bug you are trying to fix | |
| 13:53:54 | kashyap | sean-k-mooney: Keeping the first call in tact will break upstream deployments whenever using Icelake and Cascadelake | |
| 13:54:21 | sean-k-mooney[m] | the code i put in a comment in the reveiw will work for that case | |
| 13:54:25 | kashyap | sean-k-mooney: gibi: Okay: I'll keep both the checks in and wrap 'em around the workaround. | |
| 13:54:29 | sean-k-mooney[m] | and provide the corect behavior in other cases | |
| 13:54:32 | kashyap | Does that suit? | |
| 13:54:42 | sean-k-mooney[m] | have you looked at the code i provided | |
| 13:54:49 | sean-k-mooney[m] | i would perfer if you used that | |
| 13:54:59 | kashyap | Not yet; /me goes to look | |
| 13:55:54 | sean-k-mooney | my network went down for a minute or two so i read scrollback on matrix so i think im caught back up | |
| 13:56:03 | gibi | sean-k-mooney: does that change works for the *lake issue even if the compare_cpu is not replaced with the compare_hypervisor_cpu call? | |
| 13:56:37 | sean-k-mooney | oh am i ment to use the new api but i think it would yes | |
| 13:56:38 | kashyap | sean-k-mooney: So you did this: | |
| 13:56:40 | kashyap | - cpu.model = self._host.get_capabilities().host.cpu.model | |
| 13:56:40 | kashyap | + cpu.model = self._get_cpu_model_mapping(model) | |
| 13:56:51 | kashyap | sean-k-mooney: (The new API patch comes on top of the workaround.) | |
| 13:57:05 | sean-k-mooney | not just that i also did a loop over teh enabled cpu models | |
| 13:57:23 | sean-k-mooney | you added support for multiple models a few release ago | |
| 13:57:39 | sean-k-mooney | but the cpu_flags aware code path | |
| 13:57:49 | sean-k-mooney | was not updated to loop over the list of models | |
| 13:57:53 | sean-k-mooney | so that is an existing bug | |
| 13:58:04 | kashyap | Oh, wait - you wrapped it under a for loop: "for model in models:" | |
| 13:58:10 | sean-k-mooney | yes | |
| 13:58:24 | kashyap | Lemme try the tests to see what breaks; again - such a change will be done on _top_ of the workaround of both the calls | |
| 13:58:30 | sean-k-mooney | and i made sure the compare was not condtional on the flags | |
| 13:58:40 | sean-k-mooney | kashyap: no | |
| 13:58:46 | sean-k-mooney | this need to be in the first patch | |
| 13:59:31 | sean-k-mooney | well | |
| 13:59:33 | kashyap | sean-k-mooney: Well, we can't be 100% sure this works in all cases. So that's too much of a risk to take | |
| 13:59:38 | sean-k-mooney | it depend on what you mean by on top of | |
| 13:59:44 | kashyap | Because we don't have the bandwidth to test all the possibilities | |
| 13:59:50 | sean-k-mooney | if you mena later in the series then no if you mean in addtion too then yes | |
| 14:00:05 | kashyap | The safest is to wrap the 2 calls in the workaround: and _then_ do any code-flow changes on top | |
| 14:00:15 | kashyap | Does that sound reasonable? (Cc: gibi) | |
| 14:00:17 | sean-k-mooney | kashyap: as it stand i cant really supprot backporting the changes you are proposing downstream or upstream | |
| 14:00:43 | kashyap | sean-k-mooney: Please see above comment - do you see what I mean? | |
| 14:01:10 | sean-k-mooney | i do but i dont agree that is what we should do | |
| 14:02:01 | gibi | some of the risk coming from the incomplete test covarege is mitigated by having the WA flag I assume | |
| 14:02:22 | kashyap | sean-k-mooney: Shall we please arrive at a reasonable compromise, instead of "ideally situations? | |
| 14:02:25 | kashyap | gibi: Exactly | |
| 14:02:40 | sean-k-mooney | yes but i dont think we can recommend that customer use that without closing some of those gaps and qe testing | |
| 14:02:46 | kashyap | We can't possibly test all cases. So I'm not confident in any code-flow w/o absolute confidence that it's introducing unwanted regressions | |
| 14:03:20 | sean-k-mooney | kashyap: i am confident that disablinbg the check is a regession and im not comfrotabel with use telling custoemr to do that | |
| 14:03:27 | kashyap | sean-k-mooney: Addressing gaps and QE testing can come later. We take step by step based on the available bandwidth. | |
| 14:04:14 | gibi | we don't have the coverage over the current code either, and we see a bug in it for the *lake CPUs. We want to fix the bug by removing a buggy check. But by removing the buggy check we remove the good part of the check too. | |
| 14:04:59 | gibi | sean-k-mooney proposed a fix that does not remove the buggy check but improve it | |
| 14:05:10 | gibi | sure it has a testing risk as we have no coverage | |
| 14:05:50 | gibi | so the we have to decide what is bigger the lost good part of the check if we remove it or the risk by fixing the check but not having full coverage | |
| 14:05:51 | kashyap | Okay, let me try sean-k-mooney's suggestion from their comment | |
| 14:06:02 | kashyap | And "see what happens" with tests | |
| 14:06:16 | kashyap | I'll come back and comment on the change. | |
| 14:06:20 | sean-k-mooney | we can test it with the functional tests we just need to write a test for it | |
| 14:06:48 | sean-k-mooney | but i agree the test coverage is not complete and we might want to do that as a followup | |
| 14:06:53 | gibi | sean-k-mooney: we don't have the real libvirt code running in functional | |