Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-10
15:27:51 sean-k-mooney we hilighted that there nova.conf was wrong
15:27:56 gibi dansmith: thanks for the pointer I will add that to my comment in the bugreport asking tripleoo to fix the conf
15:27:57 sean-k-mooney im not sure if it has merged
15:28:02 dansmith gibi: ++
15:28:15 sean-k-mooney he was rewriting the config gen to not put the db creds in the compute nodes
15:28:37 sean-k-mooney we merged a hack to make 16.0 work but i dont know if the real fix is merged
15:28:39 sean-k-mooney ill check
15:29:13 sean-k-mooney https://review.opendev.org/#/c/718552/
15:29:14 dansmith gibi: we have another check for api.connection in conductor, which is how it knows if it's a cell or global conductor. doesn't affect compute, but just FYI for prior art
15:29:21 sean-k-mooney that is the ooo fix
15:29:27 sean-k-mooney its not merged yet
15:29:40 dansmith sean-k-mooney: ah yeah, from april.. hence my assuming it was done
15:29:47 sean-k-mooney dansmith: they used to config auto
15:29:47 dansmith 2020 has been a bit of a time warp
15:29:53 sean-k-mooney the hack was to not
15:29:56 owalsh hey...
15:30:02 dansmith sean-k-mooney: ah, heh
15:30:03 sean-k-mooney owalsh: o/
15:30:23 openstack Launchpad bug 1871482 in tripleo "nova-compute must not configure api database" [High,In progress] - Assigned to Oliver Walsh (owalsh)
15:30:23 sean-k-mooney this is the bug https://bugs.launchpad.net/tripleo/+bug/1871482
15:30:46 sean-k-mooney this was the hack https://review.opendev.org/#/c/737287/
15:30:50 sean-k-mooney which is train only
15:30:58 sean-k-mooney not on master
15:31:07 sean-k-mooney that is why tehy are hitting this now
15:31:15 sean-k-mooney victoria and train dont have it
15:31:35 sean-k-mooney *victoria and master
15:31:37 owalsh ok, they need to stop doing that. RPC version pinning is pointless with FFU
15:32:12 dansmith owalsh: note that this new failure is unrelated to pinning,
15:32:27 dansmith but to a check for service versions which fails on the compute if they have api_database.connection set on the compute
15:32:57 sean-k-mooney yep that was the root cause of the orginal issue
15:33:02 sean-k-mooney so its the same thing
15:33:17 dansmith sean-k-mooney: right, they just can't work around this new one by disabling =auto
15:33:29 sean-k-mooney ah ok
15:33:47 sean-k-mooney well i think they only get this check if we have auto set
15:33:51 dansmith no
15:33:57 sean-k-mooney no?
15:33:57 dansmith this is now done at compute startup
15:34:02 dansmith always
15:34:03 sean-k-mooney yes
15:34:39 sean-k-mooney oh instead of condtionally
15:34:41 sean-k-mooney got it
15:34:53 owalsh ok, this is master only?
15:35:00 dansmith the fail is master only,
15:35:02 sean-k-mooney so ya not a nova bug https://review.opendev.org/#/c/718552/ just need to be finished
15:35:07 dansmith we backported it to V but as a warning only
15:35:23 gibi dansmith: I'm affraid the backport also blows
15:35:28 sean-k-mooney we shoudl be backporting the ooo change in anycase
15:35:31 dansmith ohh, right right
15:35:37 dansmith because it tries and explodes before it warns
15:35:42 gibi yes
15:35:49 dansmith gibi: we could catch DBNotAllowed and warn on stable
15:35:55 gibi dansmith: yes, I will do that
15:36:11 dansmith gibi: maybe just do that on master too, to satisfy stephenfin's concern and then we can backport it cleanly
15:36:28 gibi dansmith: good idea
15:36:28 sean-k-mooney well do we need to
15:36:33 gibi thanks
15:36:39 sean-k-mooney we can but we shoudl be fixing ooo
15:36:49 sean-k-mooney they are genering an invalid config
15:36:55 dansmith sean-k-mooney: we're just talking about logging more helpful messages,
15:36:59 dansmith and avoiding a breakage on stable
15:37:04 dansmith but yes, this is a ooo thing to fix
15:37:21 gibi yes, I want to keep stable stable, or else elod will haunt me
15:37:21 sean-k-mooney yep and it was orginally a 16.1 relase blocker
15:37:38 dansmith sean-k-mooney: we backported this as a warning, but we'll explode before we warn now, if they have this config, so we need to also catch this error and warn for that too to avoid introducing a new startup fail on stable
15:37:54 gibi ^^++
15:38:10 sean-k-mooney sure that a good thing to do
15:38:31 sean-k-mooney im just going to find the downstream bz and bump it with a blocker flag request before 16.2
15:39:19 dansmith yup
15:39:59 owalsh ack, I think https://review.opendev.org/718552 is pretty much done but I wanted to run some manual tests for corner cases before removing the -W
15:42:48 stephenfin Going with a warning rather than exploding on master would be less aggressive, for sure
15:43:28 dansmith we still need to abort startup on master,
15:43:36 stephenfin yup, agreed
15:43:41 dansmith we can just catch the DB error, and log something relevant before we then abort
15:43:48 gibi yes
15:44:14 stephenfin gotcha
15:44:40 dansmith and since the error is compute specific, we can do that in utils and know that it will only happen on computes, for the log
15:44:40 stephenfin I know you already know, but the addition of the abort needs a reno
15:44:58 dansmith gibi already added one yeah?
15:45:24 dansmith https://review.opendev.org/#/c/761923/1/releasenotes/notes/warn-when-services-started-with-old-compute-fc80b4ff58a2aaea.yaml
15:45:47 dansmith this is going to abort for the same reason
15:46:05 gibi yepp, reno is there
15:46:18 gibi it is a bit different reno on the master
15:46:21 stephenfin I thought it would abort because we had API DB configuration on a compute node?
15:46:30 gibi this is the master one https://review.opendev.org/#/c/738482/10/releasenotes/notes/prevent-starting-services-with-old-compute-fc80b4ff58a2aaea.yaml
15:46:55 bauzas dansmith: just a quick question, amirite to delete https://review.opendev.org/#/c/761452/1/nova/compute/manager.py@a5115 for the RPC API 6.0 ?
15:47:35 gibi stephenfin: I can extend the reno to explain the api config issue too
15:47:36 dansmith gibi: you could catch the db exception and then do the single-cell check if you get it,
15:47:54 gibi dansmith: that is a good point too
15:47:57 dansmith gibi: avoiding the different behavior, although that won't help get people to stop configuring their computes wrong
15:48:13 stephenfin gibi: I think that would be good
15:48:15 dansmith so I guess decide whether you want to lump that in or not
15:48:33 gibi I think if the rule is not to configure api db for the compute service, then it is clearer to fail on it
15:48:40 bauzas dansmith: because even if the wallaby computes would use 5.11 (as it's the victoria API version) in case of a rolling upgrade, then they will only get objects ?
15:49:11 dansmith gibi: so, you could make this not explode, and then follow up with a separate check of just that config thing with a separate abort, separate reno, etc.. just saying you _can_ separate them I think, if you want
15:49:17 sean-k-mooney gibi: you should not configre any db creds on teh compute not just api
15:49:30 dansmith sean-k-mooney: indeed
15:49:37 sean-k-mooney since we removed the local conductor in icehouose they have not been used
15:49:45 gibi dansmith: ahh, so just split the change into two commit
15:49:47 sean-k-mooney *icehouse ish
15:49:50 gibi dansmith: I can do that

Earlier   Later