| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-26 | |||
| 13:16:36 | openstackgerrit | Elod Illes proposed openstack/os-vif stable/victoria: [stable-only] Fix lower-constraints https://review.opendev.org/c/openstack/os-vif/+/793155 | |
| 13:17:04 | elod | sean-k-mooney: this worked for me locally, let's see if zuul is also happy with it ^^^ | |
| 13:17:36 | sean-k-mooney | elod: so did the patch i pushed :) | |
| 13:17:51 | sean-k-mooney | elod: https://review.opendev.org/c/openstack/os-vif/+/792840 | |
| 13:17:58 | elod | oh did it meanwhile? :) I just saw that it failed :) | |
| 13:18:15 | sean-k-mooney | well it passed before i pushed it | |
| 13:18:24 | sean-k-mooney | it failed in teh gate because it could not install hacking | |
| 13:18:34 | sean-k-mooney | i think that was a limestone issue | |
| 13:20:10 | sean-k-mooney | elod: the recheck of min passed lower constratits for what its worth | |
| 13:20:46 | elod | it was passing before if I remember correctly | |
| 13:20:51 | sean-k-mooney | https://a23e1d5d8671587c8417-333846024fbe3cd00d8d8f9166c680e2.ssl.cf2.rackcdn.com/792840/1/check/openstack-tox-lower-constraints/f0bf4ed/ | |
| 13:21:43 | sean-k-mooney | looks like other josb are failing to conenct to opendev.org | |
| 13:22:16 | elod | the only concern why I sticked to the fixing of constraints instead of removing most of them is I think it is more "future proof" to keep more constraints. otherwise the resolver could become slower and slower. | |
| 13:22:36 | elod | but, anyway, both patch works for me :) | |
| 13:26:04 | sean-k-mooney | elod: it could although os-vif tries to have relitivly minimal deps | |
| 13:27:16 | sean-k-mooney | i would even like to reduce the set we have futher by removng oslo.i18n | |
| 13:28:00 | sean-k-mooney | we have never hadn and translsation for os-vif and i dont think we do that anymore for logs/errors in general | |
| 13:33:19 | elod | sean-k-mooney: with the minimal l-c set, if you check the job time is 10 mins for the l-c job compared to py** which are 4 mins. at least that's what I saw after the first test results | |
| 13:34:25 | sean-k-mooney | the job rund the doc generation and unit tests | |
| 13:34:39 | sean-k-mooney | the py** jobs do far less work | |
| 13:34:54 | sean-k-mooney | well maybe not far less but it is less | |
| 13:35:27 | sean-k-mooney | we shoudl compare the runtime of the two version of the patch when zuul reports back | |
| 13:35:58 | elod | of course, but with more lower constraints the resolver is quicker and takes more or less the same time. :) | |
| 13:36:16 | elod | anyway, as I said, both patch works for me :) | |
| 13:37:44 | sean-k-mooney | the ussuri version will only need a minor change to add one pep8 rule to the ignore list by the way | |
| 13:38:00 | sean-k-mooney | so im ok with fixing it in the older branch provide its simple too | |
| 13:38:37 | sean-k-mooney | that said there are some patch that for downstream reason we kind of need to merge before june 2nd | |
| 13:39:17 | sean-k-mooney | specificaly https://review.opendev.org/c/openstack/os-vif/+/765912 and https://review.opendev.org/c/openstack/os-vif/+/765970/3 | |
| 13:39:50 | gibi | sean-k-mooney, bauzas, tetsuro, melwitt: I made a step back and realized that non of you wanted a saftey check in the first place in https://review.opendev.org/c/openstack/nova-specs/+/788243 so I changed to propose not to add one. Lets see if this removes the impass | |
| 13:40:17 | gibi | impasse even | |
| 13:43:08 | sean-k-mooney | ack just finishing reviewing your qos spec. lyarwood's encypeted volumn on is next on my list then ill loop back to ^ | |
| 13:43:44 | gibi | sean-k-mooney: awesome, thanks | |
| 13:46:36 | bauzas | gibi: ack, will look again | |
| 13:59:03 | ozzzo | sean-k-mooney: I tried your grub trick and it allowed me to start the compute service with "cpu_models=Skylake-Server-IBRS,Cascadelake-Server-noTSX" | |
| 13:59:32 | ozzzo | but my cpu_info changed from "Cascadelake-Server-noTSX" to "Cascadelake-Server" whcih still doesn't match "Skylake-Server-IBRS" | |
| 13:59:59 | sean-k-mooney | they should not need to match | |
| 14:00:40 | sean-k-mooney | for live migration the xml that is used to start the dest server is a modifed one of the orginail server xml | |
| 14:00:51 | sean-k-mooney | we will not update the cpu model in that process | |
| 14:01:04 | sean-k-mooney | so the live migration will try to create the dest vm with Skylake-Server-IBRS | |
| 14:01:42 | sean-k-mooney | so you should be able to live migrate | |
| 14:02:29 | ozzzo | sean-k-mooney: I must be missing a step; I still get the "Host filter ignoring hosts" error | |
| 14:03:02 | sean-k-mooney | the host filter? that should only ignore the current host | |
| 14:03:03 | ozzzo | I did the grub trick, and then rebooted the compute, then deployed the compute | |
| 14:03:54 | sean-k-mooney | https://github.com/openstack/nova/blob/b0cd985f0c09088098f74cc0cb1df616cc0ef12b/nova/scheduler/filters/__init__.py#L22-L50 | |
| 14:04:15 | ozzzo | this is what I get in the scheduler log: https://paste.ubuntu.com/p/tX9W7dT3PH/ | |
| 14:04:57 | sean-k-mooney | actully its not that filter | |
| 14:06:05 | sean-k-mooney | thats coming form https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L490-L498 | |
| 14:06:38 | sean-k-mooney | so it means that the host has been added to the ignored host list | |
| 14:06:52 | sean-k-mooney | what release of openstack are you using by the way | |
| 14:06:56 | ozzzo | Train | |
| 14:07:16 | ozzzo | working on upgrading from centos 7-8 to prepare for train-ussuri | |
| 14:08:41 | sean-k-mooney | ok so you need to use debug logs in the schudler to see why the hsot got filtered out | |
| 14:08:51 | ozzzo | ok | |
| 14:09:20 | sean-k-mooney | something added the host to the ignored list becaue it failed a filter | |
| 14:21:32 | sean-k-mooney | gibi: im +1 on your pps spec but i have two things i woudl like to highlihgt quickly if you have time | |
| 14:22:04 | sean-k-mooney | https://review.opendev.org/c/openstack/nova-specs/+/785014/8/specs/xena/approved/qos-minimum-guaranteed-packet-rate.rst#328 and https://review.opendev.org/c/openstack/nova-specs/+/785014/8/specs/xena/approved/qos-minimum-guaranteed-packet-rate.rst#427 | |
| 14:22:41 | sean-k-mooney | gibi: neither are blocker to me i just want to make sure we are on the same page | |
| 14:23:44 | gibi | sean-k-mooney: sure | |
| 14:23:55 | gibi | sean-k-mooney: reading... | |
| 14:41:21 | ozzzo | sean-k-mooney: Here's the debug log: https://paste.ubuntu.com/p/4MrhjJs4r6/ | |
| 14:41:35 | ozzzo | at line 102 it says "Host filter ignoring hosts: chrnc-area51-os-compute-02.chtrse.com, chrnc-area51-os-compute-01.chtrse.com" | |
| 14:41:41 | ozzzo | but it's not obvious why | |
| 14:42:16 | ozzzo | compute-01 is the one where I did the grub trick | |
| 15:32:51 | stephenfin | sean-k-mooney: looks like that stable l-c patch for os-vif failed. I assume you're going to look at disabling those now, or should I do it? | |
| 15:33:52 | ozzzo | sean-k-mooney: turns out it's in the conductor log: https://paste.ubuntu.com/p/tNYS3jzXsT/ | |
| 15:34:24 | ozzzo | at 221 it says "nova.exception.MigrationPreCheckError: Migration pre-check error: CPU doesn't have compatibility." | |
| 15:34:33 | ozzzo | and then "internal error: Unknown CPU feature arch-facilities" | |
| 15:36:16 | ozzzo | actually it looks like the problem starts at line 35: | |
| 15:36:18 | ozzzo | 2021-05-26 14:30:35.457 23 DEBUG nova.conductor.tasks.live_migrate [req-d5ac7239-4326-4d9e-9731-e4fe53a19929 - - - - -] Skipping host: chrnc-area51-os-compute-02.chtrse.com because: Migration pre-check error: CPU doesn't have compatibility. | |
| 15:37:48 | sean-k-mooney | oh the arch-facilities thing | |
| 15:38:12 | ozzzo | when I google it I find this: https://bugzilla.redhat.com/show_bug.cgi?id=1658406 | |
| 15:38:12 | openstack | bugzilla.redhat.com bug 1658406 in libvirt "mode="host-model" VMs include broken "arch-facilities" flag name [libvirt]" [Urgent,Closed: errata] - Assigned to jdenemar | |
| 15:38:25 | sean-k-mooney | thats a different upgrade issue where they chagne that between centos 8 and centos 7 | |
| 15:45:56 | ozzzo | sean-k-mooney: how can I work around it? | |
| 15:46:20 | sean-k-mooney | stephenfin: we have two fixes that work | |
| 15:47:12 | sean-k-mooney | stephenfin: https://review.opendev.org/c/openstack/os-vif/+/792840 fixed lc and https://review.opendev.org/c/openstack/os-vif/+/793155 also fix it | |
| 15:47:38 | sean-k-mooney | zuul faild for other reasons | |
| 15:49:14 | sean-k-mooney | ozzzo: i think you need to update the version fo qemu then hard reboot the guest | |
| 15:49:34 | sean-k-mooney | ozzzo: i dont think there is a fix for that that does not invovle a guest reboot | |
| 15:50:25 | sean-k-mooney | ozzzo: so effectivly you cant avoid guest downtime so your simplest solution would be to cold migrate | |
| 15:51:02 | ozzzo | ok I'll experiment with cold-migration | |
| 15:51:11 | sean-k-mooney | ozzzo: unfortnetly this is a libvirt abi breakage that we cannot fix in nova | |
| 15:52:51 | ozzzo | sean-k-mooney: got it. thanks for your advice! | |
| 15:53:55 | gibi | sean-k-mooney: replied in https://review.opendev.org/c/openstack/nova-specs/+/785014 I think we are more or less on the same page | |
| 15:55:46 | sean-k-mooney | gibi: cool ill quickly skim over them one sec | |
| 15:58:07 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Repropose "vm-boot-with-unaddressed-port" spec https://review.opendev.org/c/openstack/nova-specs/+/793199 | |
| 16:02:40 | sean-k-mooney | gibi: cool i agree with youre respeonces so +w but just want to reinfoce that there is no way currently to make a neutron extention required unless nova is willign to stop supportin all neutron backends | |
| 16:03:17 | sean-k-mooney | gibi: i have brought up this problem at the last couple of ptgs | |
| 16:03:22 | gibi | sean-k-mooney: ack, then it means we will keep support for both format for a while | |
| 16:03:36 | gibi | and I will look into adding support for making someting mandatory in neutron | |
| 16:03:49 | sean-k-mooney | gibi: yes but if more them me are asking for this maybe we can actully get neutron to add a way | |
| 16:04:56 | sean-k-mooney | gibi: this type of exteniton can be implemented at the m2 2 level but since they still supprot monolity plugins the only way to make it mandatory is to implemetned above the ml2 laywer | |
| 16:05:11 | sean-k-mooney | as in move it to the core api | |
| 16:05:35 | sean-k-mooney | or start requiring that monolitich plugins implemenet a subset of the extentions | |
| 16:06:23 | bauzas | * bauzas needs to go off but I could chime on gibi's pps spec if needed | |
| 16:06:45 | sean-k-mooney | bauzas: i dont have any open issues with nova | |
| 16:06:52 | sean-k-mooney | bauzas: or the nova part of the spec | |
| 16:07:54 | sean-k-mooney | bauzas: its a neuton process/rules change that im raising | |
| 16:08:59 | bauzas | ack, will look at the spec | |
| 16:12:11 | sean-k-mooney | bauzas: i had just put a +w on it do you want me to remvoe that and leave it to you | |