| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-22 | |||
| 15:21:03 | sean-k-mooney | i dont have a bug for that currently. but i can file one | |
| 15:22:34 | sean-k-mooney | i have been tied up with a backport to queens of the in place numab rebuild that i need to get done by friday to do a backport of a feature downstream so i have not had time to work on the live migration job yet | |
| 15:29:50 | openstack | Launchpad bug 1860573 in OpenStack Compute (nova) "Nova legacy jobs should be ported to zuul v3 native jobs" [High,Triaged] | |
| 15:29:50 | sean-k-mooney | efried: https://bugs.launchpad.net/nova/+bug/1860573 | |
| 15:34:52 | efried | thanks sean-k-mooney | |
| 15:35:08 | efried | gmann: ^ for your subscribing enjoyment | |
| 16:16:15 | gmann | efried: ACK. | |
| 16:16:24 | gmann | devstack change is +A too | |
| 16:53:18 | kashyap | sean-k-mooney: Know where is the actual failure here? - https://zuul.opendev.org/t/openstack/build/ce5c1b08a48348bcad0d27dd05b0d076 | |
| 16:53:25 | kashyap | sean-k-mooney: It says "FAILED with status: 2" | |
| 16:54:02 | kashyap | Maybe this - https://0aa0d36168d68dde4230-9fa499072a9f8bf63e024cc09284603e.ssl.cf5.rackcdn.com/616603/15/check/nova-live-migration/ce5c1b0/job-output.txt | |
| 16:54:23 | kashyap | Yep | |
| 16:56:56 | efried | kashyap: n-l-m jobs are failing with PyYAML bs. | |
| 16:57:09 | efried | Fixes in devstack and devstack-gate are merging | |
| 16:57:28 | efried | https://review.opendev.org/#/c/703735/ and its dep | |
| 16:57:32 | kashyap | efried: Ah, thank you, good sir | |
| 16:58:29 | efried | if you want to see the real failure, you have to look in devstacklog.txt.gz | |
| 16:58:32 | efried | which you have to unzip | |
| 16:58:36 | efried | which is a pita | |
| 16:58:49 | efried | and will go away once we migrate n-l-m to zuulv3 | |
| 16:59:00 | efried | which is nontrivial | |
| 16:59:05 | efried | but has been for a while | |
| 16:59:05 | efried | but in the works | |
| 16:59:19 | kashyap | (Okay, I quoted you on the Nova change.) | |
| 16:59:35 | kashyap | efried: Yeah, that's the "main script" the job-output.txt moans about. Thank you | |
| 16:59:37 | efried | but we've been essentially broken by it almost continuously for a couple of weeks. | |
| 17:00:00 | kashyap | Hmm | |
| 17:01:07 | efried | kashyap: If you have any ability/time to help with that migration, it would be much appreciated. | |
| 17:01:46 | efried | I've been avoiding digging in because I would be starting off so far behind the curve that by the time I figured out how to get started someone else would (hopefully) already have it figured out. | |
| 17:02:11 | kashyap | (Afraid, not this week, preparing for a work conf on Fri) | |
| 17:02:14 | efried | not that that would be time wasted for me, but it terms of priorities, I need to be doing other things. | |
| 17:02:20 | efried | okay, it was worth a shot :P | |
| 17:02:39 | kashyap | efried: Yeah, completely understand | |
| 17:02:47 | kashyap | Can't dig into every mom-n-pop failure :D | |
| 17:03:05 | melwitt | kashyap: I'm just reminded, we'd appreciate your review on this proposed revert https://review.opendev.org/703596 | |
| 17:03:05 | kashyap | This channel is logged, /me should mind his language | |
| 17:03:11 | efried | unfortunately the mom-n-pop failures have been what's killing us, and consuming a bunch of my time. | |
| 17:03:26 | kashyap | melwitt: Hi, /me clicks | |
| 17:03:29 | efried | mostly me floundering around trying to find someone who can help with them. | |
| 17:03:34 | kashyap | efried: :-( | |
| 17:05:07 | kashyap | melwitt: To start with, that commit doesn't even tell why on god's green earth it is trying to revert | |
| 17:05:21 | melwitt | kashyap: take that up with sean-k-mooney lol | |
| 17:05:34 | kashyap | I despise "naked commit messages | |
| 17:05:41 | kashyap | s/commit/commit"/ | |
| 17:05:43 | kashyap | :D | |
| 17:05:44 | melwitt | me too | |
| 17:06:06 | sean-k-mooney | kashyap: it is what you get form the ui by default | |
| 17:06:16 | kashyap | Okay, Sean comments on PS1 | |
| 17:06:20 | sean-k-mooney | but i left some comment on the backport then fielded the reviert | |
| 17:06:36 | kashyap | sean-k-mooney: It just doesn't make sense at all to do CPU comparison check on AArch64 | |
| 17:06:45 | sean-k-mooney | that is incorrect | |
| 17:06:52 | kashyap | melwitt: I documented "why" on the main change | |
| 17:06:53 | sean-k-mooney | we have the info in /proc/cpuinfo | |
| 17:07:07 | sean-k-mooney | so we can check the cpu flags and cpu model | |
| 17:07:10 | kashyap | sean-k-mooney: I relied on the expertise on the libvirt dev who wrote that code. Want to argue with him? | |
| 17:07:19 | sean-k-mooney | sure | |
| 17:07:40 | kashyap | And hell, even AArch64 folks *themselves* told that KVM guests are to be run via 'host-passthrough' | |
| 17:07:49 | sean-k-mooney | yes | |
| 17:08:01 | kashyap | s/told/tell/ | |
| 17:08:02 | sean-k-mooney | but host-passthough means that the guest will see the host cpu flags and model | |
| 17:08:09 | kashyap | Yes, so what? | |
| 17:08:14 | sean-k-mooney | im not arguing that we should not use host-passthough | |
| 17:08:20 | kashyap | That's the whole point in this case | |
| 17:08:34 | sean-k-mooney | im am saying when you do its more important to ensur the flags and modle do not change | |
| 17:08:52 | kashyap | melwitt: sean-k-mooney: Let me quote my message from the main change, since I don't know what people have read or haven't | |
| 17:08:55 | kashyap | [quote] | |
| 17:08:56 | kashyap | So it doesn't make sense to do CPU compatability check on AArch64. And the AArch64 folks themselves recommend that the way to run KVM guests on AArch64 is via 'host-passthrough'. | |
| 17:09:00 | kashyap | It is true that libvirt does not know how to detect host CPU model on AArch64, but even if it _wants_ to know, it cannot, because even the `/proc/cpuinfo` on AArch64 doesn't show anything interesting. There are lots of vendors making different AArch64 CPUs, and they are not easily comparable. They all differ in various ways. (This is also confirmed by Jiri Denemark of libvirt.) | |
| 17:09:06 | kashyap | [/quote] | |
| 17:09:38 | kashyap | sean-k-mooney: Zooming out: what prompted you to revert? | |
| 17:10:08 | sean-k-mooney | kashyap: that may have been true when they look or on a rhel kerenl but /proc/cpuinfo has cpu Flags and the cpu modle name | |
| 17:10:27 | sean-k-mooney | however they have different capitalistation then x86 | |
| 17:10:43 | sean-k-mooney | kashyap: i saw the backport and i consider it to be a regerssion | |
| 17:11:36 | bauzas | stephenfin: so, once https://review.opendev.org/#/c/703568/ do we still have gate problems with the functional tests ? | |
| 17:12:16 | sean-k-mooney | kashyap: there have been some patches to lscpu to allow it to parse the different way the flags and model are reported on aarch64. i suspect that libvirt just need to have the same change applied to its parsing | |
| 17:12:31 | sean-k-mooney | kashyap: well actully the lscpu patch merged back in 2018 | |
| 17:19:28 | kashyap | My brain is starved of food, and my chest is aching after this bizarre bike fall. So I can't parse you fully yet | |
| 17:19:47 | kashyap | melwitt: sean-k-mooney What I know we shouldn't merge that revert, IMHO | |
| 17:20:19 | melwitt | thanks kashyap | |
| 17:20:43 | kashyap | melwitt: sean-k-mooney: Until libvirt gets the parsing correct and libvirt/QEMU CPU modelling maintainers give guidance, we should stick with current Nova behaviour. | |
| 17:21:27 | sean-k-mooney | i dissagree with that but if im being over ruled fine | |
| 17:22:30 | sean-k-mooney | with the current patch its likely that after a migration if the guess is rebooted the cpu flags and model might change | |
| 17:23:27 | sean-k-mooney | its also not clear to me if there is a risk of the guest crashign if it trys to use a instuction not supported on the dest host | |
| 17:24:26 | sean-k-mooney | given the info regarding cpu model and cpu flags is availabel in /proc/cpuinfo a correct fix in my view would have been to validate those | |
| 17:25:01 | sean-k-mooney | i walso woudl have expected a workaround config optin for this but im not planing on spending time to code that up | |
| 17:27:40 | kashyap | sean-k-mooney: All this requires extensive testing, and extremely careful audit - we haven't done any of that (yet). So we go with "solid heuristics". | |
| 17:28:22 | sean-k-mooney | solid heuristics of check nothing and hope it works? | |
| 17:28:57 | kashyap | The heuristic here being, "avoid needless checks" | |
| 17:29:44 | sean-k-mooney | we would not consider it safe to not check cpu compatabily on x86. its just seam wrong to have a lower standard for other archs | |
| 17:29:56 | sean-k-mooney | anyway i better go back to backporting | |
| 17:34:27 | sean-k-mooney | stephenfin: can you review https://review.opendev.org/#/c/701601 again when you have a chance | |
| 17:42:20 | stephenfin | sean-k-mooney: done | |
| 17:42:48 | sean-k-mooney | stephenfin: thanks | |
| 17:43:30 | sean-k-mooney | stephenfin: do you know if jan is about. he is ment to be moving this week right to start his new job next week | |
| 17:43:49 | stephenfin | bauzas: We shouldn't have, no. The issues are to do with Tempest tests, iirc (something to do with pip not uninstalling pyyaml since it was installed by the package manager) | |
| 17:44:06 | sean-k-mooney | that is being fix | |
| 17:44:07 | bauzas | all good, will recheck then | |
| 17:44:18 | stephenfin | not so fast, I don't think the fix has merged | |
| 17:44:18 | sean-k-mooney | or has been fixed | |
| 17:44:40 | stephenfin | bauzas: https://review.opendev.org/#/c/703735/ | |