| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-23 | |||
| 13:44:56 | kashyap | (As based on his suggestion is the current revision) | |
| 13:45:42 | kashyap | (And thanks for your patience so far, folks!) | |
| 13:45:55 | gibi | then I'm confused, as we discussed above that the second check alone is not enough. | |
| 13:46:07 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/870794/8/nova/virt/libvirt/driver.py#992 | |
| 13:46:14 | sean-k-mooney | kashyap: actully i dont | |
| 13:46:17 | gibi | as it does nothing when no extra flags I configured | |
| 13:46:30 | sean-k-mooney | kashyap: i said its condtionally wrong if and only if you use extra_flags | |
| 13:47:03 | sean-k-mooney | anyway i posted what i think is the correct code in the comment above ^ | |
| 13:48:16 | kashyap | sean-k-mooney: Well, I'm fine if you've refined your view _now_. But a few days ago you did write here: | |
| 13:48:19 | kashyap | (quote) | |
| 13:48:22 | kashyap | < sean-k-mooney> but if you want to future proof then sure you can put the remain check under a workaroudn | |
| 13:48:25 | kashyap | [...] | |
| 13:48:27 | kashyap | < sean-k-mooney> or just delete the first check | |
| 13:48:30 | kashyap | (/quote) | |
| 13:49:11 | kashyap | (Doing "nothing when no extra flags" are configured is still fine - as it was already explicitly tested by a downstream bug reporter in a real env) | |
| 13:49:19 | sean-k-mooney | yes i did not express my concern at that point with the looping as i tought i has already disucsst that with you | |
| 13:49:30 | kashyap | Okay. | |
| 13:49:59 | sean-k-mooney | kashyap: its not fine because its an upstream regression | |
| 13:50:00 | kashyap | gibi: sean-k-mooney: Also it was tested with the workaround as it stands: https://bugzilla.redhat.com/show_bug.cgi?id=2138381#c45 | |
| 13:50:12 | kashyap | (Unforunately, it's a private comment, so only RHTers can see it) | |
| 13:51:12 | kashyap | (Hmm, perhaps ignore that test, I think he tested it with the first revision) | |
| 13:51:25 | gibi | kashyap: you will only see the changed behaivor if the test is configuring incompatible model without extra flags. | |
| 13:51:50 | kashyap | gibi: True | |
| 13:51:55 | gibi | in the baseline that will fail the startup of nova-compute in the new code it will let the nova-compute start | |
| 13:52:08 | gibi | but fail all instance boot instead | |
| 13:52:15 | kashyap | I'm trying to find a reasonable compromise here: so shall I put both the blocks back into the code and add the workaround? | |
| 13:52:24 | kashyap | (s/blocks/calls/) | |
| 13:53:08 | kashyap | gibi: Right, failing at instance boot is still "something" - as it still provides a libvirt error. But I know we've already discussed and that's not 'acceptable' | |
| 13:53:50 | sean-k-mooney | i condier the bug you are introduceing by removign all cpu checks in the case that cpu flags is empty to be more serios then the bug you are trying to fix | |
| 13:53:54 | kashyap | sean-k-mooney: Keeping the first call in tact will break upstream deployments whenever using Icelake and Cascadelake | |
| 13:54:21 | sean-k-mooney[m] | the code i put in a comment in the reveiw will work for that case | |
| 13:54:25 | kashyap | sean-k-mooney: gibi: Okay: I'll keep both the checks in and wrap 'em around the workaround. | |
| 13:54:29 | sean-k-mooney[m] | and provide the corect behavior in other cases | |
| 13:54:32 | kashyap | Does that suit? | |
| 13:54:42 | sean-k-mooney[m] | have you looked at the code i provided | |
| 13:54:49 | sean-k-mooney[m] | i would perfer if you used that | |
| 13:54:59 | kashyap | Not yet; /me goes to look | |
| 13:55:54 | sean-k-mooney | my network went down for a minute or two so i read scrollback on matrix so i think im caught back up | |
| 13:56:03 | gibi | sean-k-mooney: does that change works for the *lake issue even if the compare_cpu is not replaced with the compare_hypervisor_cpu call? | |
| 13:56:37 | sean-k-mooney | oh am i ment to use the new api but i think it would yes | |
| 13:56:38 | kashyap | sean-k-mooney: So you did this: | |
| 13:56:40 | kashyap | + cpu.model = self._get_cpu_model_mapping(model) | |
| 13:56:40 | kashyap | - cpu.model = self._host.get_capabilities().host.cpu.model | |
| 13:56:51 | kashyap | sean-k-mooney: (The new API patch comes on top of the workaround.) | |
| 13:57:05 | sean-k-mooney | not just that i also did a loop over teh enabled cpu models | |
| 13:57:23 | sean-k-mooney | you added support for multiple models a few release ago | |
| 13:57:39 | sean-k-mooney | but the cpu_flags aware code path | |
| 13:57:49 | sean-k-mooney | was not updated to loop over the list of models | |
| 13:57:53 | sean-k-mooney | so that is an existing bug | |
| 13:58:04 | kashyap | Oh, wait - you wrapped it under a for loop: "for model in models:" | |
| 13:58:10 | sean-k-mooney | yes | |
| 13:58:24 | kashyap | Lemme try the tests to see what breaks; again - such a change will be done on _top_ of the workaround of both the calls | |
| 13:58:30 | sean-k-mooney | and i made sure the compare was not condtional on the flags | |
| 13:58:40 | sean-k-mooney | kashyap: no | |
| 13:58:46 | sean-k-mooney | this need to be in the first patch | |
| 13:59:31 | sean-k-mooney | well | |
| 13:59:33 | kashyap | sean-k-mooney: Well, we can't be 100% sure this works in all cases. So that's too much of a risk to take | |
| 13:59:38 | sean-k-mooney | it depend on what you mean by on top of | |
| 13:59:44 | kashyap | Because we don't have the bandwidth to test all the possibilities | |
| 13:59:50 | sean-k-mooney | if you mena later in the series then no if you mean in addtion too then yes | |
| 14:00:05 | kashyap | The safest is to wrap the 2 calls in the workaround: and _then_ do any code-flow changes on top | |
| 14:00:15 | kashyap | Does that sound reasonable? (Cc: gibi) | |
| 14:00:17 | sean-k-mooney | kashyap: as it stand i cant really supprot backporting the changes you are proposing downstream or upstream | |
| 14:00:43 | kashyap | sean-k-mooney: Please see above comment - do you see what I mean? | |
| 14:01:10 | sean-k-mooney | i do but i dont agree that is what we should do | |
| 14:02:01 | gibi | some of the risk coming from the incomplete test covarege is mitigated by having the WA flag I assume | |
| 14:02:22 | kashyap | sean-k-mooney: Shall we please arrive at a reasonable compromise, instead of "ideally situations? | |
| 14:02:25 | kashyap | gibi: Exactly | |
| 14:02:40 | sean-k-mooney | yes but i dont think we can recommend that customer use that without closing some of those gaps and qe testing | |
| 14:02:46 | kashyap | We can't possibly test all cases. So I'm not confident in any code-flow w/o absolute confidence that it's introducing unwanted regressions | |
| 14:03:20 | sean-k-mooney | kashyap: i am confident that disablinbg the check is a regession and im not comfrotabel with use telling custoemr to do that | |
| 14:03:27 | kashyap | sean-k-mooney: Addressing gaps and QE testing can come later. We take step by step based on the available bandwidth. | |
| 14:04:14 | gibi | we don't have the coverage over the current code either, and we see a bug in it for the *lake CPUs. We want to fix the bug by removing a buggy check. But by removing the buggy check we remove the good part of the check too. | |
| 14:04:59 | gibi | sean-k-mooney proposed a fix that does not remove the buggy check but improve it | |
| 14:05:10 | gibi | sure it has a testing risk as we have no coverage | |
| 14:05:50 | gibi | so the we have to decide what is bigger the lost good part of the check if we remove it or the risk by fixing the check but not having full coverage | |
| 14:05:51 | kashyap | Okay, let me try sean-k-mooney's suggestion from their comment | |
| 14:06:02 | kashyap | And "see what happens" with tests | |
| 14:06:16 | kashyap | I'll come back and comment on the change. | |
| 14:06:20 | sean-k-mooney | we can test it with the functional tests we just need to write a test for it | |
| 14:06:48 | sean-k-mooney | but i agree the test coverage is not complete and we might want to do that as a followup | |
| 14:06:53 | gibi | sean-k-mooney: we don't have the real libvirt code running in functional | |
| 14:07:16 | gibi | we have the libvirt/driver but we don't have the compareCPU from libvirt itself | |
| 14:07:19 | sean-k-mooney | we dont but we could enhance the fixture if we really wanted | |
| 14:07:41 | sean-k-mooney | i was thinking if we could test this in ci via tempest but we dont really contole the enve enought to do that reliably | |
| 14:08:01 | sean-k-mooney | so i dont really want to go down that route | |
| 14:08:11 | gibi | yeah, best would be to test this with real hardver and the TSX case but that is haaard | |
| 14:08:34 | gibi | (should be possible downstream though) | |
| 14:09:29 | sean-k-mooney | with whitebox we coudl proably do it | |
| 14:09:46 | kashyap | I'm running the UTs with Sean's suggestion on top this: https://review.opendev.org/c/openstack/nova/+/870794 | |
| 14:10:02 | sean-k-mooney | how we woululd do it upstream in tempest is selct a cpu_model that we knwo is unaviable | |
| 14:10:11 | kashyap | (If that gives some confidence in a day or so. I can wrap it in the workaround.) | |
| 14:10:12 | sean-k-mooney | and disable the cpu flag to make it look like nehalem | |
| 14:10:37 | sean-k-mooney | the problem is that a provider could add that model in teh future | |
| 14:10:54 | sean-k-mooney | the best way to do that would be to use a model that requests a remvoed feature like TSX | |
| 14:11:01 | sean-k-mooney | and disable tsk via the extra flags | |
| 14:11:39 | sean-k-mooney | presumable our ci provireder also dont have tsx avaible anymore | |
| 14:11:48 | sean-k-mooney | if they have been updating htere kernels/microcode | |
| 14:12:08 | kashyap | Yeah, we don't know that, and can't rely on it. | |
| 14:15:31 | gibi | so my position is that let's take the risk and doing sean-k-mooney's fix but have the WA flag added. So if the change is good then all is golden, but if the change has some unforseen side effect then we can turn it off while we improve it. For me the risk of this scenario is acceptable. | |