| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-08-14 | |||
| 17:03:11 | sean-k-mooney | i think its the call to the conductor that trigering the save with the refeshed content | |
| 17:03:47 | sean-k-mooney | artom: fyi ^ | |
| 17:04:26 | sean-k-mooney | we could try moving the refresh to here https://review.opendev.org/c/openstack/nova/+/710848/6/nova/compute/manager.py#2371 | |
| 17:04:53 | sean-k-mooney | which i think would result in it being saved here instead https://review.opendev.org/c/openstack/nova/+/710848/6/nova/compute/manager.py#2260 | |
| 17:05:28 | sean-k-mooney | which would be better i think | |
| 17:05:42 | sean-k-mooney | can you try that locally | |
| 17:05:52 | artom | sean-k-mooney, I saw the comments but since I didn't write the actual patch I have to start from scratch to figure out what's going on | |
| 17:06:41 | sean-k-mooney | ok im going to finish in 5 mins but ill try to just move that locally and see if it passes your test case | |
| 17:37:02 | sean-k-mooney | ... we have a seperate sqlalchemy 2.0 issue | |
| 17:44:18 | opendevreview | sean mooney proposed openstack/nova master: test fixing this when allocation fails. https://review.opendev.org/c/openstack/nova/+/926350 | |
| 17:44:34 | sean-k-mooney | artom: ^ passes locally if i run the funciontal tests | |
| 17:44:40 | sean-k-mooney | but the serise also need a rebase | |
| 17:45:18 | sean-k-mooney | ... | |
| 17:45:30 | sean-k-mooney | or you know i could not do that on patchset 6... | |
| 17:45:49 | artom | Yeah, that relation chain is... special :) | |
| 17:48:20 | opendevreview | sean mooney proposed openstack/nova master: test fixing this when allocation fails. https://review.opendev.org/c/openstack/nova/+/926350 | |
| 17:49:41 | sean-k-mooney | ok that should be better | |
| 17:50:07 | sean-k-mooney | gibi: ^ i think that is a better fix | |
| 17:50:29 | sean-k-mooney | ... or it would be if it didnt fail | |
| 17:53:46 | sean-k-mooney | ok im not sure why that didnt work but it did not so im just not goign to look at that today | |
| 17:54:38 | opendevreview | Merged openstack/nova master: Skip new image format tests https://review.opendev.org/c/openstack/nova/+/926215 | |
| 17:55:04 | sean-k-mooney | oh actully i think i do no | |
| 17:55:14 | sean-k-mooney | that not where the excption is being raised | |
| 17:59:10 | opendevreview | sean mooney proposed openstack/nova master: test fixing this when allocation fails. https://review.opendev.org/c/openstack/nova/+/926350 | |
| 17:59:25 | sean-k-mooney | ^ actully passes locally | |
| 18:06:41 | sean-k-mooney | artom: gibi i have left some comments to explain why i think this also work and is cleaner but also why the patch i have is still not ideal | |
| #openstack-nova - 2024-08-15 | |||
| 01:24:41 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with SEV-ES memory encryption https://review.opendev.org/c/openstack/nova/+/926106 | |
| 01:26:30 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with SEV-ES memory encryption https://review.opendev.org/c/openstack/nova/+/926106 | |
| 08:40:05 | gibi | sean-k-mooney: artom: I'm still in the process of understanding the logic of the relationship between the instance.pci_devices and the pci_tracker state. At some point the acconting of the two got out of sync and I think that is the place we need to fix | |
| 08:40:15 | gibi | the refresh is just a band aid | |
| 09:03:18 | zigo | bauzas: Hi there! Are you aware of this problem? https://bugs.launchpad.net/nova/+bug/2077070 | |
| 09:03:37 | zigo | Is Sylvain in holidays btw? :) | |
| 09:13:55 | frickler | isn't all of france in holiday during august? but this sounds related to what sean-k-mooney and gibi were discussing earlier at least in as much as pci devices are affected | |
| 09:44:36 | gibi | sean-k-mooney: artom: https://review.opendev.org/c/openstack/nova/+/710848/13#message-354946be57b39eb88b67872230c226361d8a3559 this is where I am at. The resource trackker correctly removes the device from the instance.pci_devices list during the aborting of the claim, but that update is lost when we return to the compute manager | |
| 09:49:49 | sean-k-mooney[m] | zigo is the compute node marked as deleted | |
| 09:49:53 | sean-k-mooney[m] | we dont actully delete the row | |
| 09:50:16 | gibi | sean-k-mooney: artom: yeah Claim ctor clones the instance https://github.com/openstack/nova/blob/a7c82399b237aff44810ac76b0c4d10416c3bdf9/nova/compute/claims.py#L65-L66 | |
| 09:51:14 | sean-k-mooney[m] | gibi: im going to gab a coffee but ill be back in 5 mins and ill take a look | |
| 10:01:33 | sean-k-mooney | ok actull | |
| 10:03:34 | gibi | abort claim logic working on a copy of the instance so the cleanup of the instance is lost when nova returns to the compute manager | |
| 10:04:04 | sean-k-mooney | do you want to to jump on a google meet and debug this in 10 mins. that sound plausible yes looking at the clone. | |
| 10:04:16 | gibi | sure I can | |
| 10:04:27 | sean-k-mooney | i belive we do that to make sure we dont modify the instnace object during the claim | |
| 10:04:43 | sean-k-mooney | btu i guess we did in one path and didnt in another | |
| 10:04:48 | sean-k-mooney | and the two are out of sync | |
| 10:05:13 | gibi | should we wait for artom? | |
| 10:05:19 | gibi | maybe he is interested too | |
| 10:05:21 | sean-k-mooney | we can | |
| 10:05:52 | gibi | then I will grab a quick lunch, and let you look at what I found in the meantime | |
| 10:06:04 | gibi | and we can jump on a call when artom is arong | |
| 10:06:07 | gibi | arond | |
| 10:06:11 | sean-k-mooney | i didnt sleep well last night (didnt get to sleep until 4am) so im just startign my day now so wanted to skim my email first while having my morning coffee | |
| 10:06:22 | sean-k-mooney | so we can chat when they are onlien | |
| 10:06:33 | gibi | OK | |
| 10:15:11 | sean-k-mooney | frickler: and yes that is a differnt topic. we have a very old bug, with an eaislly old bugfix that "works" but its fixing the sideefect rather then the cause and we are tryign to under stand what is the cuase and how to move forward | |
| 10:16:05 | zigo | sean-k-mooney: I'm not sure, I haven't checked for that, you're probably right that it just stays as deleted, but my bug still stand. | |
| 10:16:30 | sean-k-mooney | zigo: i have not looked at your bug yet :) | |
| 10:17:04 | sean-k-mooney | openstack compute service delete <uuid> is deleting the comptue service not the compute node by the way | |
| 10:17:18 | sean-k-mooney | it may do one as a side effect of the other but those are two logically differnt things | |
| 10:17:59 | sean-k-mooney | zigo: are there any instance on the compute node at the time your deleting the compute service | |
| 10:18:10 | sean-k-mooney | espically any that are using pci devices | |
| 10:18:43 | zigo | sean-k-mooney: I had a case where that was the issue, but in another case, no, there was no instance at all, even less with a PCI device attached. | |
| 10:20:31 | sean-k-mooney | zigo so reinstalling a node because of a hardware issue would not normally invovle deleting the comppute service | |
| 10:20:35 | sean-k-mooney | is there a reason you did that | |
| 10:20:54 | sean-k-mooney | im not saying we should not look into the clean up | |
| 10:21:12 | zigo | It's just our procedure to clean-up nova, cinder and neutron services when we decomission a node... | |
| 10:21:17 | sean-k-mooney | but the procudure yoru takign is not what i would expect/recommend if you are replacing a node bug keeping the host name | |
| 10:21:30 | zigo | Ah. | |
| 10:21:56 | sean-k-mooney | zigo this is what i woudl expect if you were chanign the hostname i.e scale in and out | |
| 10:22:12 | sean-k-mooney | but not if you are preserving the host name | |
| 10:23:22 | sean-k-mooney | i think that might be why we dont see this downstream | |
| 10:24:16 | sean-k-mooney | we have 2 diffent but related procedures. if its just a hardware failure and you want to just swap out the failed hardware then we woudl expct you do phsyicall do that and then just reinstalll but not delete the compute service | |
| 10:24:50 | sean-k-mooney | for hosts where the instances are on cpeh for example that allows replacement of the failed hardware even if you have not evacuated the instances although that is prefered | |
| 10:25:25 | sean-k-mooney | in our old instally scalining in and out a new server ended up with a new hostname | |
| 10:25:35 | sean-k-mooney | so it would not hit this edgecases | |
| 10:25:58 | zigo | sean-k-mooney: I would expect that I can do what I want, the way I want, without experiencing such a bug ... :) | |
| 10:29:50 | sean-k-mooney | well you can expect that :) | |
| 11:21:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926407 | |
| 13:40:41 | sean-k-mooney | gibi: i think what you are proposing is a better fix then mine as you have moved the cleanup to the correct function | |
| 13:40:57 | sean-k-mooney | gibi: but i left a comment inline. | |
| 13:42:47 | sean-k-mooney | im not sure we will have time to sync with artom beofre our internal meeting start so we might just do that after we finish wiht them or we can do it async | |
| 13:43:16 | sean-k-mooney | and check back tomorow | |
| 13:53:39 | gibi | let's try tomorrow. I don't think I will have the mental capacity after our internal call. But I will clean up the unit and functional test failures at least in the meantime | |
| 13:54:01 | gibi | also I agree about free_instance_claims but I will check for fallouts | |
| 13:54:34 | artom | sean-k-mooney, gibi, what's up? | |
| 13:54:57 | sean-k-mooney | artom: gibi foudn the root cause of the bug | |
| 13:55:19 | artom | Sweet | |
| 13:55:48 | gibi | artom: this is the distilled version https://review.opendev.org/c/openstack/nova/+/926407 | |
| 13:56:12 | gibi | the longer form is in the comments of https://review.opendev.org/c/openstack/nova/+/710848/13#message-354946be57b39eb88b67872230c226361d8a3559 | |
| 13:57:09 | artom | Oh wow | |
| 13:57:11 | gibi | I have still 30 mins befor the next call so if you artom, sean-k-mooney want to sync on it then I can explain in gmeet | |
| 13:57:11 | sean-k-mooney | artom: i was suggeting this morning that we coudl hop on a call and try and find it together but we said we woudl wait for you to come online before doing that and in the mean time gibi made some progress | |
| 13:57:34 | artom | That is so insidious, well found! | |
| 13:58:09 | artom | Of course then the fear is - what else are we affecting, and potentially breaking? | |
| 13:58:15 | sean-k-mooney | im ment ot have a 1:1 in 3 minute but not sure if its going to happen | |
| 13:58:21 | sean-k-mooney | artom: right | |
| 13:58:31 | artom | I feel like our test coverage is enough at this point that's it's _probably_ fine | |
| 13:58:47 | sean-k-mooney | maybe | |
| 13:59:04 | gibi | artom: exactly. I also a bit affraid of the potential fallout | |
| 13:59:20 | gibi | the functional test is clean (exept one test case that I can explain and fix) | |