Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-10
13:13:57 lyarwood ah
13:14:02 lyarwood right kk
13:14:14 sean-k-mooney we when with train when we tought the other qemu detach thing was a focal issue
13:14:17 lyarwood and yeah either way the multinode grenade change isn't simple
13:14:21 sean-k-mooney that gives us 5.2.0
13:14:33 sean-k-mooney on tain i think
13:14:34 lyarwood even calling the old scripts is borked as it assumes we are using devstack-gate etc
13:14:51 lyarwood sean-k-mooney: ack
13:16:25 lyarwood http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/ I don't see Victoria listed here however
13:18:12 kashyap stephenfin: Really nice rework here - https://review.opendev.org/#/c/756551/
13:18:26 kashyap (Also the commit message :))
13:19:19 sean-k-mooney i kind fo feel like using tabels in a commit is cheating but ya it explains things well
13:30:59 kashyap sean-k-mooney: Hehe; what else would you use?
13:31:37 kashyap In my books, it's perfectly fair game to see tables in a commit message :)
13:42:44 stephenfin gibi: Have you seen the comment on https://review.opendev.org/#/c/738482/ ?
13:46:12 stephenfin gibi: There's a bug report filed for it here https://bugs.launchpad.net/tripleo/+bug/1903655
13:46:12 openstack Launchpad bug 1903655 in tripleo "Compute component jobs in master branch are failing with ERROR nova nova.exception.DBNotAllowed: nova-compute attempted direct database access which is not allowed by policy" [Critical,Triaged]
13:50:53 gibi stephenfin: thanks for the notification, I haven't seen it
13:50:57 gibi yet
13:58:29 gibi I have to be on a call, but after It I will look into it
14:11:48 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: WIP - Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284
14:13:15 lyarwood ^ reviews welcome on that spec now btw, still WIP but hopefully ready for serious reviews. I'll have PoC code updated this week once I've written the func tests.
14:56:44 sean-k-mooney that does a lot
14:57:10 sean-k-mooney im guessing the duplicaiton is a result of that but havent looke at the test for it in a long time
15:16:32 gibi stephenfin dansmith: I've looked into https://bugs.launchpad.net/tripleo/+bug/1903655 and it seems that tripleoo configures [api_database]/connection for the nova-compute service and that causes that the service version check assumes that we are in a top level controller service which can access the api database https://github.com/openstack/nova/blob/master/nova/utils.py#L1064-L1072
15:16:33 openstack Launchpad bug 1903655 in tripleo "Compute component jobs in master branch are failing with ERROR nova nova.exception.DBNotAllowed: nova-compute attempted direct database access which is not allowed by policy" [Critical,Triaged]
15:17:08 dansmith gibi: yeah I saw your analysis and I'm sure you're right
15:17:27 gibi is ther a smarter way to decided if we are inside a cell?
15:17:28 dansmith we have other such checks I think, so this is probably just the first time they've hit something fatal to even notice
15:17:37 dansmith no, I think this is a good thing
15:17:59 dansmith although, hmm
15:18:14 dansmith er, yeah, this is just compute that's failing
15:18:25 dansmith so yeah, I think this is good
15:18:34 gibi same could happen with a cell controller as well
15:18:38 gibi I mean a cell conductor
15:18:48 dansmith yep, but that's fine
15:19:29 dansmith a cell conductor shouldn't be, but can be configured with api_connection (for the last few reasons why you might need it), but it'll just survey the whole cluster instead of the cell, but that should be fine
15:20:08 gibi I see, the remaining upcalls
15:20:13 dansmith right
15:22:30 stephenfin gibi: That sounds reasonable. What's the remedy though? I assume we can't simply say "don't do this", at least not without a deprecation window?
15:22:32 gibi OK then now I feel better about asking tripleoo folks to check and fix there nova-cpu.conf
15:23:13 gibi stephenfin: if this is an invalid config then nova-compute is allowed to fail on it
15:23:15 dansmith gibi: definitely. we don't deploy it that way in devstack, and I'm pretty sure we've had and fixed this issue downstream
15:23:25 dansmith I'm surprised the upstream tripleo is still doing it
15:23:38 stephenfin Okay, can we add a check to nova-compute startup to hard fail on that config being present?
15:23:42 stephenfin So the error is more obvious?
15:23:47 dansmith sure
15:24:05 gibi stephenfin: that is a good point, but I think the current service level check runs first and fail first
15:24:23 gibi stephenfin: so I cannot add this to the _inithost
15:24:29 gibi or I can but it is pointless
15:24:44 gibi buuut I might be able add someting to the cmd
15:25:27 dansmith so, we have another such check in rpcapi, and a catch for them doing it wrongly, with a warning to explain
15:26:02 dansmith https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L439-L457
15:26:25 dansmith so, I'm guessing they've also not configured upgrade_levels=auto,
15:26:29 dansmith else they would be exploding there too
15:26:52 gibi could be, my tripleoo knowledge is close to 0
15:27:18 sean-k-mooney i think we have a down stream bug for this
15:27:23 dansmith point being, this is not a new behavior or requirement
15:27:28 sean-k-mooney owalsh was fixing it
15:27:42 dansmith sean-k-mooney: I thought he did a while ago, which is why I'm surprised we're still seeing it
15:27:51 sean-k-mooney we hilighted that there nova.conf was wrong
15:27:56 gibi dansmith: thanks for the pointer I will add that to my comment in the bugreport asking tripleoo to fix the conf
15:27:57 sean-k-mooney im not sure if it has merged
15:28:02 dansmith gibi: ++
15:28:15 sean-k-mooney he was rewriting the config gen to not put the db creds in the compute nodes
15:28:37 sean-k-mooney we merged a hack to make 16.0 work but i dont know if the real fix is merged
15:28:39 sean-k-mooney ill check
15:29:13 sean-k-mooney https://review.opendev.org/#/c/718552/
15:29:14 dansmith gibi: we have another check for api.connection in conductor, which is how it knows if it's a cell or global conductor. doesn't affect compute, but just FYI for prior art
15:29:21 sean-k-mooney that is the ooo fix
15:29:27 sean-k-mooney its not merged yet
15:29:40 dansmith sean-k-mooney: ah yeah, from april.. hence my assuming it was done
15:29:47 dansmith 2020 has been a bit of a time warp
15:29:47 sean-k-mooney dansmith: they used to config auto
15:29:53 sean-k-mooney the hack was to not
15:29:56 owalsh hey...
15:30:02 dansmith sean-k-mooney: ah, heh
15:30:03 sean-k-mooney owalsh: o/
15:30:23 sean-k-mooney this is the bug https://bugs.launchpad.net/tripleo/+bug/1871482
15:30:23 openstack Launchpad bug 1871482 in tripleo "nova-compute must not configure api database" [High,In progress] - Assigned to Oliver Walsh (owalsh)
15:30:46 sean-k-mooney this was the hack https://review.opendev.org/#/c/737287/
15:30:50 sean-k-mooney which is train only
15:30:58 sean-k-mooney not on master
15:31:07 sean-k-mooney that is why tehy are hitting this now
15:31:15 sean-k-mooney victoria and train dont have it
15:31:35 sean-k-mooney *victoria and master
15:31:37 owalsh ok, they need to stop doing that. RPC version pinning is pointless with FFU
15:32:12 dansmith owalsh: note that this new failure is unrelated to pinning,
15:32:27 dansmith but to a check for service versions which fails on the compute if they have api_database.connection set on the compute
15:32:57 sean-k-mooney yep that was the root cause of the orginal issue
15:33:02 sean-k-mooney so its the same thing
15:33:17 dansmith sean-k-mooney: right, they just can't work around this new one by disabling =auto
15:33:29 sean-k-mooney ah ok
15:33:47 sean-k-mooney well i think they only get this check if we have auto set
15:33:51 dansmith no
15:33:57 dansmith this is now done at compute startup
15:33:57 sean-k-mooney no?
15:34:02 dansmith always
15:34:03 sean-k-mooney yes
15:34:39 sean-k-mooney oh instead of condtionally

Earlier   Later