| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-10 | |||
| 12:16:49 | sean-k-mooney | oh you removed the parent of the grenade job by mistake but https://review.opendev.org/#/c/742056/ corrects it and movs to v3 | |
| 12:31:56 | sean-k-mooney | lyarwood: could we make the multinode grenade job nonvoting until https://review.opendev.org/#/c/742056/ is merged | |
| 12:32:45 | sean-k-mooney | i think we also will need to use the cloud archive to get libvirt 6.0 for some of the other stable branches | |
| 12:32:53 | sean-k-mooney | that have to run on bionic | |
| 12:34:28 | sean-k-mooney | am i correct in assuming ussuri proably has support for blockdev too or was that added in victoria | |
| 12:34:47 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1901739/comments/6 is the root cause right? | |
| 12:34:47 | openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 12:40:10 | sean-k-mooney | i think im going to repopose https://opendev.org/openstack/devstack/commit/7f7f488bc385dd707a3a6d8dae7859bbe72182e5 with victoria instead | |
| 13:06:42 | kashyap | sean-k-mooney: Yeah; the workaround is mentioned in the bug as a comment | |
| 13:07:13 | sean-k-mooney | using libvirt 6.0.0 | |
| 13:07:25 | kashyap | Yep | |
| 13:07:36 | sean-k-mooney | really that just means we are not fixing the issue form openstack and raising the min libvirt | |
| 13:07:41 | sean-k-mooney | which is not really a good thing | |
| 13:07:45 | kashyap | sean-k-mooney: It's using the legacy "-drive" approach; and the modern one ("-blockdev") should fix it | |
| 13:08:00 | kashyap | sean-k-mooney: It's not an OpenStack issue | |
| 13:08:07 | sean-k-mooney | yep i know | |
| 13:08:22 | sean-k-mooney | and i also know we cant force libvirt to only use drive or blockdev | |
| 13:08:30 | sean-k-mooney | which is why we can workaournd it form nova | |
| 13:09:12 | sean-k-mooney | my point is for all deployments that cant use libvirt 6 there is no way for them to work around this | |
| 13:09:28 | sean-k-mooney | well excpet upgrade | |
| 13:09:45 | sean-k-mooney | anmyway i think the victoia cloud archive had 6.0.0 | |
| 13:10:04 | kashyap | Yeah; the whole backports / how far back should upstream support is a tricky thing | |
| 13:10:07 | sean-k-mooney | so im going to bump the version we use in devstack on the older branches | |
| 13:10:25 | kashyap | The answer is: "if you want such backported fixes", use an "enterprise" distro | |
| 13:10:35 | kashyap | (So goes the argument) | |
| 13:10:35 | sean-k-mooney | well no | |
| 13:10:49 | sean-k-mooney | the anser is that libvirt could actully maintain branches and do backports | |
| 13:10:57 | kashyap | Well, they do that | |
| 13:10:59 | sean-k-mooney | they dont which forces distros to do it | |
| 13:11:09 | kashyap | But how you have to backport is an upstream decision | |
| 13:11:27 | sean-k-mooney | yep its just a strange one | |
| 13:11:27 | kashyap | They maintain several "stable" branches | |
| 13:11:39 | sean-k-mooney | very few work the way they do | |
| 13:11:50 | sean-k-mooney | anyway this is simple tweek | |
| 13:12:03 | sean-k-mooney | since it packaged in the uca | |
| 13:12:26 | kashyap | I never claimed it's all perfect :) | |
| 13:12:53 | sean-k-mooney | if it means i dont have to keep rechecking stuff then thats good enough | |
| 13:13:11 | lyarwood | sorry just back from lunch | |
| 13:13:33 | sean-k-mooney | lyarwood: im assuming the zuulv3 patch will take a while to merge | |
| 13:13:45 | lyarwood | hmm that's a good point about the UCA, why isn't the grenade bionic job using it? | |
| 13:13:49 | sean-k-mooney | lyarwood:so im just going to swap to victoria uca which has 6.0.0 | |
| 13:13:56 | sean-k-mooney | it is but train | |
| 13:13:57 | lyarwood | ah | |
| 13:14:02 | lyarwood | right kk | |
| 13:14:14 | sean-k-mooney | we when with train when we tought the other qemu detach thing was a focal issue | |
| 13:14:17 | lyarwood | and yeah either way the multinode grenade change isn't simple | |
| 13:14:21 | sean-k-mooney | that gives us 5.2.0 | |
| 13:14:33 | sean-k-mooney | on tain i think | |
| 13:14:34 | lyarwood | even calling the old scripts is borked as it assumes we are using devstack-gate etc | |
| 13:14:51 | lyarwood | sean-k-mooney: ack | |
| 13:16:25 | lyarwood | http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/ I don't see Victoria listed here however | |
| 13:18:12 | kashyap | stephenfin: Really nice rework here - https://review.opendev.org/#/c/756551/ | |
| 13:18:26 | kashyap | (Also the commit message :)) | |
| 13:19:19 | sean-k-mooney | i kind fo feel like using tabels in a commit is cheating but ya it explains things well | |
| 13:30:59 | kashyap | sean-k-mooney: Hehe; what else would you use? | |
| 13:31:37 | kashyap | In my books, it's perfectly fair game to see tables in a commit message :) | |
| 13:42:44 | stephenfin | gibi: Have you seen the comment on https://review.opendev.org/#/c/738482/ ? | |
| 13:46:12 | stephenfin | gibi: There's a bug report filed for it here https://bugs.launchpad.net/tripleo/+bug/1903655 | |
| 13:46:12 | openstack | Launchpad bug 1903655 in tripleo "Compute component jobs in master branch are failing with ERROR nova nova.exception.DBNotAllowed: nova-compute attempted direct database access which is not allowed by policy" [Critical,Triaged] | |
| 13:50:53 | gibi | stephenfin: thanks for the notification, I haven't seen it | |
| 13:50:57 | gibi | yet | |
| 13:58:29 | gibi | I have to be on a call, but after It I will look into it | |
| 14:11:48 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: WIP - Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284 | |
| 14:13:15 | lyarwood | ^ reviews welcome on that spec now btw, still WIP but hopefully ready for serious reviews. I'll have PoC code updated this week once I've written the func tests. | |
| 14:56:44 | sean-k-mooney | that does a lot | |
| 14:57:10 | sean-k-mooney | im guessing the duplicaiton is a result of that but havent looke at the test for it in a long time | |
| 15:16:32 | gibi | stephenfin dansmith: I've looked into https://bugs.launchpad.net/tripleo/+bug/1903655 and it seems that tripleoo configures [api_database]/connection for the nova-compute service and that causes that the service version check assumes that we are in a top level controller service which can access the api database https://github.com/openstack/nova/blob/master/nova/utils.py#L1064-L1072 | |
| 15:16:33 | openstack | Launchpad bug 1903655 in tripleo "Compute component jobs in master branch are failing with ERROR nova nova.exception.DBNotAllowed: nova-compute attempted direct database access which is not allowed by policy" [Critical,Triaged] | |
| 15:17:08 | dansmith | gibi: yeah I saw your analysis and I'm sure you're right | |
| 15:17:27 | gibi | is ther a smarter way to decided if we are inside a cell? | |
| 15:17:28 | dansmith | we have other such checks I think, so this is probably just the first time they've hit something fatal to even notice | |
| 15:17:37 | dansmith | no, I think this is a good thing | |
| 15:17:59 | dansmith | although, hmm | |
| 15:18:14 | dansmith | er, yeah, this is just compute that's failing | |
| 15:18:25 | dansmith | so yeah, I think this is good | |
| 15:18:34 | gibi | same could happen with a cell controller as well | |
| 15:18:38 | gibi | I mean a cell conductor | |
| 15:18:48 | dansmith | yep, but that's fine | |
| 15:19:29 | dansmith | a cell conductor shouldn't be, but can be configured with api_connection (for the last few reasons why you might need it), but it'll just survey the whole cluster instead of the cell, but that should be fine | |
| 15:20:08 | gibi | I see, the remaining upcalls | |
| 15:20:13 | dansmith | right | |
| 15:22:30 | stephenfin | gibi: That sounds reasonable. What's the remedy though? I assume we can't simply say "don't do this", at least not without a deprecation window? | |
| 15:22:32 | gibi | OK then now I feel better about asking tripleoo folks to check and fix there nova-cpu.conf | |
| 15:23:13 | gibi | stephenfin: if this is an invalid config then nova-compute is allowed to fail on it | |
| 15:23:15 | dansmith | gibi: definitely. we don't deploy it that way in devstack, and I'm pretty sure we've had and fixed this issue downstream | |
| 15:23:25 | dansmith | I'm surprised the upstream tripleo is still doing it | |
| 15:23:38 | stephenfin | Okay, can we add a check to nova-compute startup to hard fail on that config being present? | |
| 15:23:42 | stephenfin | So the error is more obvious? | |
| 15:23:47 | dansmith | sure | |
| 15:24:05 | gibi | stephenfin: that is a good point, but I think the current service level check runs first and fail first | |
| 15:24:23 | gibi | stephenfin: so I cannot add this to the _inithost | |
| 15:24:29 | gibi | or I can but it is pointless | |
| 15:24:44 | gibi | buuut I might be able add someting to the cmd | |
| 15:25:27 | dansmith | so, we have another such check in rpcapi, and a catch for them doing it wrongly, with a warning to explain | |
| 15:26:02 | dansmith | https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L439-L457 | |
| 15:26:25 | dansmith | so, I'm guessing they've also not configured upgrade_levels=auto, | |
| 15:26:29 | dansmith | else they would be exploding there too | |
| 15:26:52 | gibi | could be, my tripleoo knowledge is close to 0 | |
| 15:27:18 | sean-k-mooney | i think we have a down stream bug for this | |
| 15:27:23 | dansmith | point being, this is not a new behavior or requirement | |