| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-04 | |||
| 16:35:34 | kashyap | gibi: Just posted on #virt, OFTC network. | |
| 16:36:18 | kashyap | gibi: Is this blocking patch merges? | |
| 16:39:05 | melwitt | I've been struggling for a couple of days trying to get an approved patch through the gate, but I'm not sure whether that particular bug is involved. I would need to re-look at the logs to verify | |
| 16:39:25 | kashyap | (I've reposted the looping libvirtd log bits as a plain text, as the pastebins expire) | |
| 16:40:58 | kashyap | melwitt: Noted; Michael, the reporter, was saying on last Friday that it's "intermittent", which makes it a bit more difficult to debug | |
| 16:41:56 | melwitt | yeah, that's been the theme of all of the gate bugs I'm aware of. intermittent and thus hard to troubleshoot :( | |
| 16:42:06 | melwitt | *current gate bugs I'm aware of | |
| 16:43:01 | kashyap | Yeah, matches my past experience | |
| 16:43:53 | kashyap | melwitt: In the same vein as how Twitter (I'm not on it) seems to label Trump's tweets as misleading, wonder we should adapt that text for these intermittent bugs :D | |
| 16:44:26 | kashyap | - Some or all of the content shared in this Tweet is disputed and might be misleading about an election or other civic process. | |
| 16:44:29 | kashyap | + Some or all of the content shared in this bug is disputed and might be misleading due to intermittent failures. | |
| 16:46:02 | kashyap | gibi: melwitt: More seriously, can I "subscribe" (Cc) someone else to a LaunchPad, right? | |
| 16:46:21 | kashyap | IIRC, yes. /me tries | |
| 16:46:26 | melwitt | I think you can | |
| 16:48:11 | kashyap | melwitt: I can't :-( I wanted to Cc Michal from libvirt but it says "No items matched <email ID>" | |
| 16:48:44 | melwitt | do you know his launchpad id? | |
| 16:48:55 | kashyap | melwitt: Oh, having a Launchpad ID is mandatory? | |
| 16:49:19 | melwitt | it might be, that's the only way I've seen subscribing | |
| 16:50:44 | kashyap | Ah, noted. I don't think he has one - searching doesn't show up anything. | |
| 16:51:41 | kashyap | melwitt: I pointed to him on IRC; he's taking a look | |
| 16:53:00 | melwitt | thanks! | |
| 16:55:23 | kashyap | melwitt: gibi: That's quick -- Michal (Privoznik) says it looks like a genuine bug. I'll update the bug once we get more details | |
| 16:56:35 | melwitt | sounds great, thank you kashyap | |
| 17:06:24 | kashyap | melwitt: So, the libvirt version in the logs above is 5.4.0; but havne't we switche dalready to libvirt-6.0.0? | |
| 17:06:39 | kashyap | lyarwood: --^ (By "we", I mean upstream CI) | |
| 17:08:11 | kashyap | So Michal says, there were improvements in libvirt-6.1.0 release on this area of event loops. | |
| 17:13:52 | stephenfin | lyarwood: Could you cast an eye over https://review.opendev.org/#/c/631053/ this evening, please? | |
| 17:14:04 | stephenfin | It's been around for quite a while :-D | |
| 17:18:01 | mloza | hello, is it possible to update the video model to vmga of an existing instance? | |
| 17:19:11 | mloza | if i edit /etc/libvirt/qemu/instance-, it reverts to default when the instance is hard rebooted | |
| 17:20:04 | stephenfin | mloza: Outside of rebuilding to a new image, no. We don't support setting it via the flavor so resize isn't an option | |
| 17:21:47 | stephenfin | mloza: You'll have to modify the DB manually if you want to avoid the rebuild | |
| 17:22:18 | mloza | can you tell me which table do I need modify | |
| 17:22:23 | mloza | to modif* | |
| 17:24:03 | stephenfin | iirc, we persist image metadata properties for an image in the instance_system_metadata table | |
| 17:25:16 | stephenfin | in case it wasn't obvious, back up the DB first and note that any support guarantees are gone out the window if you modify the DB manually | |
| 17:37:27 | sean-k-mooney | i think the table name is system_metadata not instance_system metadata but yes we do | |
| 17:37:34 | sean-k-mooney | with an img_ prefix | |
| 17:38:04 | sean-k-mooney | os if it was hw_video_model it woudl be img_hw_video_model in the db | |
| 17:38:48 | sean-k-mooney | mloza: were you asking about this on the mailing list too? we basically said the same in our replies | |
| 17:49:45 | bauzas | gibi: stephenfin: fwiw, you accepted a breaking RPC change with https://review.opendev.org/#/c/715326/29/nova/compute/manager.py@3327 by not accepting a nullable accels argument | |
| 17:50:06 | bauzas | sean-k-mooney: ^ | |
| 17:52:41 | bauzas | if a compute client is sending a 5.0 cast to a compute service, then there won't have a accels argument, so the manager will return an exception | |
| 17:56:26 | dansmith | bauzas: good catch | |
| 17:56:39 | dansmith | bauzas: not too late to fix that | |
| 17:57:44 | bauzas | dansmith: I wonder whether we should fix it by the compute v5 proxy I write or having another change we could backport to victoria ? | |
| 17:58:00 | dansmith | bauzas: another change that we backport | |
| 17:58:09 | bauzas | ack, doing it then | |
| 17:58:19 | dansmith | before people try to upgrade to victoria | |
| 17:58:52 | dansmith | technically, this shouldn't be a problem if people upgrade their controllers first, but if they don't, they'll get an explosion that won't be easy to decipher | |
| 17:58:59 | bauzas | yeah | |
| 17:59:01 | dansmith | well, no, actuall,y | |
| 17:59:16 | dansmith | it would blow up for anyone with an old compute if the version is pinned | |
| 17:59:21 | bauzas | given most of the operators upgrade first their conductors, it shouldn't be a problem | |
| 17:59:33 | bauzas | but in case they pin it, yes | |
| 17:59:44 | dansmith | no, because the client was done properly, it will break | |
| 17:59:52 | dansmith | even if they don't pin, assuming they use =auto | |
| 17:59:56 | bauzas | if you pin the API, right? | |
| 18:00:07 | bauzas | why then for auto ? | |
| 18:00:18 | dansmith | the default is =auto, which will select the lowest version supported by all computes, | |
| 18:00:24 | bauzas | ahah | |
| 18:00:32 | dansmith | so if you have one old compute, api, conductor, etc will all choose the older and will not send that argument, | |
| 18:00:33 | bauzas | I see | |
| 18:00:35 | dansmith | and thus it'll explode | |
| 18:00:49 | bauzas | TIL | |
| 18:00:54 | bauzas | about how auto works | |
| 18:00:59 | dansmith | so s/good catch/great catch/ :) | |
| 18:01:21 | dansmith | easy backport to fix it though, luckily, before people start rolling to V | |
| 18:01:28 | bauzas | yup, writing it now | |
| 18:01:36 | bauzas | git stash first tho :) | |
| 18:01:46 | dansmith | heh yeah I bet :) | |
| 18:04:47 | bauzas | dansmith: do we have some documentation about pin=auto ? maybe on your blog ? | |
| 18:04:53 | dansmith | lol | |
| 18:05:07 | dansmith | I guess I didn't think it was really something people often confused | |
| 18:05:25 | dansmith | I'd expect the config doc to be accurate, but let me look | |
| 18:05:53 | bauzas | nevermind, found it ;) https://docs.openstack.org/nova/latest/user/upgrade.html | |
| 18:06:51 | dansmith | the config doc says "don't worry, we'll handle it" without any detail | |
| 18:07:07 | bauzas | yeah, looking at the config option text | |
| 18:07:30 | dansmith | but honestly, | |
| 18:08:00 | dansmith | from the perspective of a deployer, they really just need to know that auto selects the right version, | |
| 18:08:01 | dansmith | without needing to know that our client needs *a* version and not per-target | |
| 18:08:17 | dansmith | we definitely don't want to look up each target's version before *each* call because that would suck | |
| 18:08:22 | bauzas | yeah | |
| 18:08:37 | dansmith | so maybe some developer docs to explain how it actually works, if you think that's important, but ... | |
| 18:08:42 | bauzas | but if we have problem with the default, then that's a high bug for rolling upgrades | |
| 18:09:50 | bauzas | grmblbl, I need to install both python37 and pre_commit module just for a single git commit | |
| 18:10:24 | dansmith | well, | |
| 18:10:33 | dansmith | not just the default, but also if someone were to hand-set the right version | |
| 18:11:28 | bauzas | yup, any rolling upgrade | |
| 18:20:27 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Bump the Compute RPC API to version 6.0 https://review.opendev.org/761452 | |
| 18:27:41 | openstack | Launchpad bug 1902925 in OpenStack Compute (nova) victoria "Upgrades to compute RPC API 5.12 are broken" [Critical,Confirmed] - Assigned to Sylvain Bauza (sylvain-bauza) | |
| 18:27:41 | bauzas | dansmith: triaged https://bugs.launchpad.net/nova/+bug/1902925 lemme say if I told something wrong | |
| 18:29:03 | dansmith | bauzas: yep, sounds good.. it makes it sound like the error was combined with the rpc client being right and the server being wrong, | |
| 18:29:13 | dansmith | but even if the client hadn't bumped the version, it would still be wrong | |
| 18:29:17 | dansmith | but yeah, what you said is all right | |
| 18:29:22 | bauzas | ack | |
| 18:29:45 | bauzas | the fix is simple, but I wonder whether we already have a test suite somewhere for it | |
| 18:30:51 | dansmith | well, generally there should be a test to pass the old version and new version of the call, | |
| 18:31:01 | dansmith | but I think it requires adding two test cases, which probably didn't happen | |
| 19:09:46 | bauzas | can anyone know how to get the full stacktrace returned by the API when we have a HTTP500 ? | |