Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-08-15
10:17:59 sean-k-mooney zigo: are there any instance on the compute node at the time your deleting the compute service
10:18:10 sean-k-mooney espically any that are using pci devices
10:18:43 zigo sean-k-mooney: I had a case where that was the issue, but in another case, no, there was no instance at all, even less with a PCI device attached.
10:20:31 sean-k-mooney zigo so reinstalling a node because of a hardware issue would not normally invovle deleting the comppute service
10:20:35 sean-k-mooney is there a reason you did that
10:20:54 sean-k-mooney im not saying we should not look into the clean up
10:21:12 zigo It's just our procedure to clean-up nova, cinder and neutron services when we decomission a node...
10:21:17 sean-k-mooney but the procudure yoru takign is not what i would expect/recommend if you are replacing a node bug keeping the host name
10:21:30 zigo Ah.
10:21:56 sean-k-mooney zigo this is what i woudl expect if you were chanign the hostname i.e scale in and out
10:22:12 sean-k-mooney but not if you are preserving the host name
10:23:22 sean-k-mooney i think that might be why we dont see this downstream
10:24:16 sean-k-mooney we have 2 diffent but related procedures. if its just a hardware failure and you want to just swap out the failed hardware then we woudl expct you do phsyicall do that and then just reinstalll but not delete the compute service
10:24:50 sean-k-mooney for hosts where the instances are on cpeh for example that allows replacement of the failed hardware even if you have not evacuated the instances although that is prefered
10:25:25 sean-k-mooney in our old instally scalining in and out a new server ended up with a new hostname
10:25:35 sean-k-mooney so it would not hit this edgecases
10:25:58 zigo sean-k-mooney: I would expect that I can do what I want, the way I want, without experiencing such a bug ... :)
10:29:50 sean-k-mooney well you can expect that :)
11:21:30 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926407
13:40:41 sean-k-mooney gibi: i think what you are proposing is a better fix then mine as you have moved the cleanup to the correct function
13:40:57 sean-k-mooney gibi: but i left a comment inline.
13:42:47 sean-k-mooney im not sure we will have time to sync with artom beofre our internal meeting start so we might just do that after we finish wiht them or we can do it async
13:43:16 sean-k-mooney and check back tomorow
13:53:39 gibi let's try tomorrow. I don't think I will have the mental capacity after our internal call. But I will clean up the unit and functional test failures at least in the meantime
13:54:01 gibi also I agree about free_instance_claims but I will check for fallouts
13:54:34 artom sean-k-mooney, gibi, what's up?
13:54:57 sean-k-mooney artom: gibi foudn the root cause of the bug
13:55:19 artom Sweet
13:55:48 gibi artom: this is the distilled version https://review.opendev.org/c/openstack/nova/+/926407
13:56:12 gibi the longer form is in the comments of https://review.opendev.org/c/openstack/nova/+/710848/13#message-354946be57b39eb88b67872230c226361d8a3559
13:57:09 artom Oh wow
13:57:11 gibi I have still 30 mins befor the next call so if you artom, sean-k-mooney want to sync on it then I can explain in gmeet
13:57:11 sean-k-mooney artom: i was suggeting this morning that we coudl hop on a call and try and find it together but we said we woudl wait for you to come online before doing that and in the mean time gibi made some progress
13:57:34 artom That is so insidious, well found!
13:58:09 artom Of course then the fear is - what else are we affecting, and potentially breaking?
13:58:15 sean-k-mooney im ment ot have a 1:1 in 3 minute but not sure if its going to happen
13:58:21 sean-k-mooney artom: right
13:58:31 artom I feel like our test coverage is enough at this point that's it's _probably_ fine
13:58:47 sean-k-mooney maybe
13:59:04 gibi artom: exactly. I also a bit affraid of the potential fallout
13:59:20 gibi the functional test is clean (exept one test case that I can explain and fix)
13:59:22 sean-k-mooney so what i was hoping is you could integrate the changes form gibi and try an fix the test coverage as needed
13:59:38 sean-k-mooney then we could see what that looks liek tomorrow
14:00:07 gibi sean-k-mooney: artom: I can clean up my patch so artom can focus on the socket id thing
14:00:17 sean-k-mooney ack that works too
14:00:24 gibi and we can ask melwitt or dansmith for review as a second core
14:01:20 sean-k-mooney speaking of mel i need to loop back to here unified limits changes
14:01:33 sean-k-mooney melwitt: did the test change to oslo.limits land yet
14:02:21 sean-k-mooney melwitt: oh ok you abandoned https://review.opendev.org/c/openstack/oslo.limit/+/924024
14:02:49 sean-k-mooney so i guess the intent is to proceed without that for now in https://review.opendev.org/c/openstack/nova/+/924025
14:04:03 sean-k-mooney melwitt: i have not looked at that in a while but i see you still have it marked as work in progress in gerrit
14:04:32 sean-k-mooney are you happy the current version is ready for review or should i loop back in a few days
14:50:17 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926407
15:00:16 melwitt sean-k-mooney: yeah the intent was to go for a nova-only approach. and oops, not supposed to be marked as WIP anymore, I'll unmark them
#openstack-nova - 2024-08-16
03:27:30 opendevreview melanie witt proposed openstack/nova master: Add [workarounds]unified_limits_permissive_enforcement https://review.opendev.org/c/openstack/nova/+/924025
06:53:38 opendevreview melanie witt proposed openstack/nova master: Add [workarounds]unified_limits_permissive_enforcement https://review.opendev.org/c/openstack/nova/+/924025
08:25:39 gibi artom: sean-k-mooney: bauzas: you probably remember when we looked at why unshelving to an ironic host fails with the cryptic error as Destination.node is not set while unshelving to a libvirt host works even if the compute api only sets Destination.host there too. The answer is in https://bugs.launchpad.net/nova/+bug/2075504/comments/3
08:26:04 gibi basically the scheduler fills Destination.node based on .host if the mapping is 1:1
08:45:28 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926407
10:00:47 sean-k-mooney[m] i didnt think shelve was supported for ironic
10:01:28 sean-k-mooney[m] i know unselve could work but i would expect that to not be supproted as it would require chaning the flavor
10:01:51 sean-k-mooney[m] since ironic nodes should bnot have cpu, ram or disk
10:05:15 sean-k-mooney[m] and should be using a flavor with only a custom resource class
10:05:15 sean-k-mooney[m] so placement should retrun no allocation for the host
10:09:30 gibi sean-k-mooney: yes, it is not supported but it fails in a very stange way
10:09:44 gibi so I have that cosmetic bug open to fail it in a more clear way
10:10:20 gibi and while we discussed that we had an open question how don't unshelve to libvirt node does not fail with the same reason, the missing Destination.node field
10:10:25 sean-k-mooney[m] there is also a tempet/job config bug. there is a temest setting for pining which host is used for the source and dest node for migrations
10:10:26 gibi and yesterday I found the reason
10:10:48 sean-k-mooney[m] we either are not setting it and shelve tests dont respect it or we need to add that to the job config
10:10:54 gibi sean-k-mooney: yeah, also James is fixing the host selection logic for unshelve to host
10:11:00 gibi in tempest
10:11:00 sean-k-mooney[m] that or we need to add a shelve sepcific config for that
10:11:09 sean-k-mooney[m] ok
10:11:50 sean-k-mooney[m] we should proably return an exception in the 1:many case
10:12:35 gibi yeah that would probably make sense otherwise the scheduler blows up later anyway
10:13:03 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926407
12:54:39 opendevreview Takashi Natsume proposed openstack/nova-specs master: Create specs directory for 2025.1 Epoxy https://review.opendev.org/c/openstack/nova-specs/+/924227
13:31:19 opendevreview sean mooney proposed openstack/nova master: split monkey_patching form import https://review.opendev.org/c/openstack/nova/+/922425
13:31:20 opendevreview sean mooney proposed openstack/nova master: refactor cmd dir for non eventlet console_scripts https://review.opendev.org/c/openstack/nova/+/904424
13:31:20 opendevreview sean mooney proposed openstack/nova master: Remove nova debugger funcitonality https://review.opendev.org/c/openstack/nova/+/922496
13:31:21 opendevreview sean mooney proposed openstack/nova master: Add IO Thread Pool Executor https://review.opendev.org/c/openstack/nova/+/922497
13:31:21 opendevreview sean mooney proposed openstack/nova master: remove eventlet.tpool https://review.opendev.org/c/openstack/nova/+/905287
13:31:22 opendevreview sean mooney proposed openstack/nova master: use futureist for scatter_gather https://review.opendev.org/c/openstack/nova/+/905284
13:31:22 opendevreview sean mooney proposed openstack/nova master: use futerist for RBD calls https://review.opendev.org/c/openstack/nova/+/917962
13:31:57 sean-k-mooney stephenfin: gibi: ^ rebased and adress the current out standing issues if you have time to review
13:32:09 stephenfin aclk
13:55:26 opendevreview sean mooney proposed openstack/nova master: add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015
13:55:27 opendevreview sean mooney proposed openstack/nova master: add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844
13:55:27 opendevreview sean mooney proposed openstack/nova master: add healthcheck tracker to nova context https://review.opendev.org/c/openstack/nova/+/829468
13:55:28 opendevreview sean mooney proposed openstack/nova master: add healthcheck endpoint to proxy commands https://review.opendev.org/c/openstack/nova/+/830703
13:55:28 opendevreview sean mooney proposed openstack/nova master: [WIP] add iniital health check to compute agent https://review.opendev.org/c/openstack/nova/+/906911
13:55:29 opendevreview sean mooney proposed openstack/nova master: [WIP] add libvirt connection healtcheck https://review.opendev.org/c/openstack/nova/+/907424
13:55:47 sean-k-mooney thats ^ just a rebase i need to go though them and see what review feedback is present
14:10:29 opendevreview Balazs Gibizer proposed openstack/nova master: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926407
14:34:11 opendevreview Zhang Hua proposed openstack/nova master: BlockDeviceMapping object has no attribute copy https://review.opendev.org/c/openstack/nova/+/920374
15:07:06 opendevreview Arnaud Morin proposed openstack/nova master: libvirt: Log exception if failure checking block job https://review.opendev.org/c/openstack/nova/+/907720
16:27:00 gibi artom: sean-k-mooney: the running CI looking green on https://review.opendev.org/c/openstack/nova/+/926407 and I fixed up all the comments.
16:27:24 gibi that is my last contribution for this week and next week I'm off.
16:27:43 gibi see you on the 26th

Earlier   Later