| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-09 | |||
| 12:42:14 | sean-k-mooney | if you are replacing the node and keeping the same identity/hostname | |
| 12:42:27 | sean-k-mooney | then you shoudl not do a compute service delete | |
| 12:42:31 | sean-k-mooney | as part of the replacement | |
| 12:43:01 | sean-k-mooney | and you shoudl take the old compute_node uuid and put it in the compute node file | |
| 12:43:06 | tobias-urdin | yes automation/manual steps failed for removing the nova service record (and thus RP) before installing new node, but issue would be caught if nova-compute failed start so that it reported as down | |
| 12:43:28 | sean-k-mooney | right but what im saying is the logic is wrong | |
| 12:43:44 | sean-k-mooney | there are 2 ways to do that replacement | |
| 12:44:13 | sean-k-mooney | you can scale in (fully deleting the node in nova/neutron/placemetn) and then scale out | |
| 12:44:23 | sean-k-mooney | if you do that there is a strict ordering requirement | |
| 12:44:34 | sean-k-mooney | the name cannot be reused until its fully deleted | |
| 12:44:49 | sean-k-mooney | or you can replace the node without deleting the compute/neutron agent services | |
| 12:45:14 | sean-k-mooney | which is how we normally recommend you replace a node when maintaining its identity | |
| 12:45:40 | tobias-urdin | normally we just delete everything but this time somebody forgot the compute service delete step, i would like to protect operators from having a nova-compute reporting up when it's not working | |
| 12:46:28 | sean-k-mooney | ... | |
| 12:46:34 | tobias-urdin | fail on nova-compute startup if RP creation failed | |
| 12:46:50 | tobias-urdin | like neutron-ovs-agent is looping forever and not reporting up if br-int or local_ip is wrong | |
| 12:47:07 | tobias-urdin | safety first ¯\_(ツ)_/¯ | |
| 12:47:14 | sean-k-mooney | so the problem is that we expect to be able to start the compute agents before placement is up | |
| 12:47:29 | sean-k-mooney | and for them to eventually register when it becomes aviableable | |
| 12:47:55 | sean-k-mooney | that woudl still be true if you have something liek systemd | |
| 12:48:00 | sean-k-mooney | that will restart nova-compute externally | |
| 12:48:13 | tobias-urdin | but if we explicitly fail on placement RP conflict then? would that work | |
| 12:48:14 | sean-k-mooney | so your change might be ok | |
| 12:48:42 | sean-k-mooney | it woudl only work if you have an external service restarting it | |
| 12:49:20 | sean-k-mooney | im not nessisarly agaisnt this change | |
| 12:50:10 | sean-k-mooney | im a little wary of just doing it for any exception | |
| 12:51:27 | sean-k-mooney | tobias-urdin: the up status of a compute service is primarly ment to signal that its able to comunicate via rpc with the hypervior | |
| 12:51:48 | sean-k-mooney | i say it that way as we have specific logic to mark it down if the connection to libivrt is lost | |
| 12:52:06 | sean-k-mooney | its technially not ment to be a healtcheck | |
| 12:52:37 | sean-k-mooney | so im a little concerned about scope creap | |
| 12:53:26 | sean-k-mooney | with that said you are stoping the agent if it fails | |
| 12:53:43 | sean-k-mooney | so the meaning of up technically has not been changed by this patch | |
| 12:54:37 | tobias-urdin | yeah i understand, for us basically since we don't verify anything in placement specifically with monitoring, the nova-compute was up but it never scheduled any workload there | |
| 12:55:06 | tobias-urdin | it's a messy edge case but if we get ResourceProviderCreationFailed and it's a conflict we could perhaps just fail on startup to not report up | |
| 12:55:18 | sean-k-mooney | for what its worht this is why im working on adding a per process healt check endpoint | |
| 12:55:26 | tobias-urdin | so took us some day or two before we noticed it | |
| 12:55:44 | tobias-urdin | yea | |
| 12:55:51 | sean-k-mooney | so i think i would be happier if you added a new except for ResourceProviderCreationFailed | |
| 12:55:58 | sean-k-mooney | rather then doing this for just Exception | |
| 12:56:16 | sean-k-mooney | the other thing im thinking about is | |
| 12:56:23 | sean-k-mooney | how does your chage work with ironic | |
| 12:57:05 | sean-k-mooney | im not expictly aware of an issue with ironic but each ironic agent is managing multiple compute nodes (1 per ironic node) each with there own RP | |
| 12:58:26 | sean-k-mooney | in its current form if there was an error for any of the ironic RPs it would cause the ironic compute agent to exit | |
| 12:58:43 | sean-k-mooney | so we may want to skip this if its ironic | |
| 13:01:30 | sean-k-mooney | tobias-urdin: what release are you runing by the ay | |
| 13:01:34 | sean-k-mooney | *way | |
| 13:05:11 | tobias-urdin | mix of yoga and zed, but nova is yoga | |
| 13:05:28 | sean-k-mooney | ack so ya you dont have the stabel uuid feature | |
| 13:05:34 | sean-k-mooney | which would catch this. | |
| 13:06:03 | tobias-urdin | didn't think about the ironic use-case | |
| 13:06:18 | tobias-urdin | would nova-compute service be down in that case? | |
| 13:07:11 | sean-k-mooney | so usign the normal meaning of up/down it should be up if it cna comunicate with teh ironic rest api and it accessabel over the rpc bus | |
| 13:07:32 | sean-k-mooney | even if one of the ironic computes is not usabel | |
| 13:07:42 | sean-k-mooney | comptue nodes do not have an up or down state | |
| 13:07:47 | sean-k-mooney | only comptue services do | |
| 13:08:01 | tobias-urdin | sorry, with stable uuid feature | |
| 13:08:18 | sean-k-mooney | oh for stable uuid we explcitly dont use the fature with ironic | |
| 13:08:40 | sean-k-mooney | we put if checks in the code to skip it if its ironic because ironic is special | |
| 13:08:56 | tobias-urdin | i mean if we were on newer release with stable uuid feature would nova-compute service fail to start and not report itself as up if there was a conflict like this? | |
| 13:08:56 | sean-k-mooney | it will be less special once the peer_list and hashring is removed | |
| 13:09:08 | sean-k-mooney | yes | |
| 13:09:24 | sean-k-mooney | in antelope it shoudl not start for libvirt | |
| 13:09:59 | sean-k-mooney | to reuse the same hostname you now need to create /var/lib/nova/comptue_node with the old comptue node uuid | |
| 13:10:14 | tobias-urdin | then imo i will just drop the patches and ignore the issue until we are there, it's a edge case that at best helps operators know they have done something wrong (which imo is good, but prob not worth the work here) | |
| 13:10:25 | sean-k-mooney | the placment RP is curated useing the hypervior_hostname as the RP name and the compute node uuid as the rp uuid | |
| 13:10:39 | sean-k-mooney | i prefer your second patch by the way | |
| 13:10:56 | sean-k-mooney | i woudl not be against that approch as a backprot pre antelope | |
| 13:11:12 | sean-k-mooney | if it also didnt raise for ironic | |
| 13:13:00 | tobias-urdin | ack, thx for your time! | |
| 13:22:00 | opendevreview | Merged openstack/nova stable/xena: Unify placement client singleton implementations https://review.opendev.org/c/openstack/nova/+/858999 | |
| 13:31:56 | opendevreview | Merged openstack/nova stable/zed: add a regression test for all compute RPCAPI 6.x pinnings for rebuild https://review.opendev.org/c/openstack/nova/+/900307 | |
| 13:41:01 | opendevreview | Rajesh Tailor proposed openstack/nova-specs master: List requested availability zones https://review.opendev.org/c/openstack/nova-specs/+/904368 | |
| 13:44:20 | ykarel | hi is the issue known checked https://bugs.launchpad.net/nova/+bug/2045785 ? | |
| 14:33:48 | sean-k-mooney | ykarel: maybe a timeout issue with the tempet test | |
| 14:34:02 | sean-k-mooney | i knwo there is at least one unfixed tempest bug | |
| 14:34:38 | sean-k-mooney | where tempet is not currectly using the instance action events api to determin if voluem detach? has completed | |
| 14:34:49 | sean-k-mooney | so perhaps this test is not waiting properly | |
| 15:01:32 | dansmith | bauzas: FYI, I have a dentist appointment so I'll miss the meeting | |
| 15:01:44 | bauzas | ++ hope you'll be all good :) | |
| 15:58:23 | kashyap | dansmith: Good luck with the dentist :) | |
| 16:00:59 | bauzas | #startmeeting nova | |
| 16:00:59 | opendevmeet | Meeting started Tue Jan 9 16:00:59 2024 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:59 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:59 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:01:06 | bauzas | heyho | |
| 16:01:14 | bauzas | happy new year folks ! | |
| 16:01:18 | elodilles | o/ | |
| 16:01:23 | grandchild | o/ | |
| 16:01:24 | elodilles | happy new year o/ | |
| 16:01:41 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:02:53 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | |
| 16:02:56 | fwiesel | o/ | |
| 16:03:16 | kashyap | bauzas: I had one more item to bring up, but didn't add it to the wiki (bad me) | |
| 16:03:27 | bauzas | ok let's start our first meeting of 0x7E8 | |
| 16:03:44 | bauzas | kashyap: no worries, I'll ping you at the end | |
| 16:03:50 | bauzas | #topic Bugs (stuck/critical) | |
| 16:03:55 | bauzas | #info No Critical bug | |
| 16:03:58 | bauzas | woohoo | |
| 16:04:02 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 43 new untriaged bugs (-1 since the last meeting) | |
| 16:04:09 | bauzas | thanks folks ! | |