| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-26 | |||
| 12:40:21 | sean-k-mooney | and stable wallaby is since it has your fix | |
| 12:40:31 | sean-k-mooney | so it would be victoria and older | |
| 12:40:35 | gibi | I agree with stephenfin | |
| 12:40:48 | stephenfin | sean-k-mooney: victoria and older wfm | |
| 12:41:32 | sean-k-mooney | ok i just reject the victoria fix if that fails to install hacking again or fails in general ill drop it and then propose the other patches for stable and rebase any pending patches | |
| 12:41:52 | sean-k-mooney | well the 2 i want to backport in anycase :) | |
| 13:07:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document virtio-net multiqueue https://review.opendev.org/c/openstack/nova/+/792362 | |
| 13:07:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add 'hw:vif_multiqueue_enabled' flavor extra spec https://review.opendev.org/c/openstack/nova/+/792356 | |
| 13:07:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move 'hw:pmu', 'hw_pmu' parsing to nova.virt.hardware https://review.opendev.org/c/openstack/nova/+/792364 | |
| 13:16:36 | openstackgerrit | Elod Illes proposed openstack/os-vif stable/victoria: [stable-only] Fix lower-constraints https://review.opendev.org/c/openstack/os-vif/+/793155 | |
| 13:17:04 | elod | sean-k-mooney: this worked for me locally, let's see if zuul is also happy with it ^^^ | |
| 13:17:36 | sean-k-mooney | elod: so did the patch i pushed :) | |
| 13:17:51 | sean-k-mooney | elod: https://review.opendev.org/c/openstack/os-vif/+/792840 | |
| 13:17:58 | elod | oh did it meanwhile? :) I just saw that it failed :) | |
| 13:18:15 | sean-k-mooney | well it passed before i pushed it | |
| 13:18:24 | sean-k-mooney | it failed in teh gate because it could not install hacking | |
| 13:18:34 | sean-k-mooney | i think that was a limestone issue | |
| 13:20:10 | sean-k-mooney | elod: the recheck of min passed lower constratits for what its worth | |
| 13:20:46 | elod | it was passing before if I remember correctly | |
| 13:20:51 | sean-k-mooney | https://a23e1d5d8671587c8417-333846024fbe3cd00d8d8f9166c680e2.ssl.cf2.rackcdn.com/792840/1/check/openstack-tox-lower-constraints/f0bf4ed/ | |
| 13:21:43 | sean-k-mooney | looks like other josb are failing to conenct to opendev.org | |
| 13:22:16 | elod | the only concern why I sticked to the fixing of constraints instead of removing most of them is I think it is more "future proof" to keep more constraints. otherwise the resolver could become slower and slower. | |
| 13:22:36 | elod | but, anyway, both patch works for me :) | |
| 13:26:04 | sean-k-mooney | elod: it could although os-vif tries to have relitivly minimal deps | |
| 13:27:16 | sean-k-mooney | i would even like to reduce the set we have futher by removng oslo.i18n | |
| 13:28:00 | sean-k-mooney | we have never hadn and translsation for os-vif and i dont think we do that anymore for logs/errors in general | |
| 13:33:19 | elod | sean-k-mooney: with the minimal l-c set, if you check the job time is 10 mins for the l-c job compared to py** which are 4 mins. at least that's what I saw after the first test results | |
| 13:34:25 | sean-k-mooney | the job rund the doc generation and unit tests | |
| 13:34:39 | sean-k-mooney | the py** jobs do far less work | |
| 13:34:54 | sean-k-mooney | well maybe not far less but it is less | |
| 13:35:27 | sean-k-mooney | we shoudl compare the runtime of the two version of the patch when zuul reports back | |
| 13:35:58 | elod | of course, but with more lower constraints the resolver is quicker and takes more or less the same time. :) | |
| 13:36:16 | elod | anyway, as I said, both patch works for me :) | |
| 13:37:44 | sean-k-mooney | the ussuri version will only need a minor change to add one pep8 rule to the ignore list by the way | |
| 13:38:00 | sean-k-mooney | so im ok with fixing it in the older branch provide its simple too | |
| 13:38:37 | sean-k-mooney | that said there are some patch that for downstream reason we kind of need to merge before june 2nd | |
| 13:39:17 | sean-k-mooney | specificaly https://review.opendev.org/c/openstack/os-vif/+/765912 and https://review.opendev.org/c/openstack/os-vif/+/765970/3 | |
| 13:39:50 | gibi | sean-k-mooney, bauzas, tetsuro, melwitt: I made a step back and realized that non of you wanted a saftey check in the first place in https://review.opendev.org/c/openstack/nova-specs/+/788243 so I changed to propose not to add one. Lets see if this removes the impass | |
| 13:40:17 | gibi | impasse even | |
| 13:43:08 | sean-k-mooney | ack just finishing reviewing your qos spec. lyarwood's encypeted volumn on is next on my list then ill loop back to ^ | |
| 13:43:44 | gibi | sean-k-mooney: awesome, thanks | |
| 13:46:36 | bauzas | gibi: ack, will look again | |
| 13:59:03 | ozzzo | sean-k-mooney: I tried your grub trick and it allowed me to start the compute service with "cpu_models=Skylake-Server-IBRS,Cascadelake-Server-noTSX" | |
| 13:59:32 | ozzzo | but my cpu_info changed from "Cascadelake-Server-noTSX" to "Cascadelake-Server" whcih still doesn't match "Skylake-Server-IBRS" | |
| 13:59:59 | sean-k-mooney | they should not need to match | |
| 14:00:40 | sean-k-mooney | for live migration the xml that is used to start the dest server is a modifed one of the orginail server xml | |
| 14:00:51 | sean-k-mooney | we will not update the cpu model in that process | |
| 14:01:04 | sean-k-mooney | so the live migration will try to create the dest vm with Skylake-Server-IBRS | |
| 14:01:42 | sean-k-mooney | so you should be able to live migrate | |
| 14:02:29 | ozzzo | sean-k-mooney: I must be missing a step; I still get the "Host filter ignoring hosts" error | |
| 14:03:02 | sean-k-mooney | the host filter? that should only ignore the current host | |
| 14:03:03 | ozzzo | I did the grub trick, and then rebooted the compute, then deployed the compute | |
| 14:03:54 | sean-k-mooney | https://github.com/openstack/nova/blob/b0cd985f0c09088098f74cc0cb1df616cc0ef12b/nova/scheduler/filters/__init__.py#L22-L50 | |
| 14:04:15 | ozzzo | this is what I get in the scheduler log: https://paste.ubuntu.com/p/tX9W7dT3PH/ | |
| 14:04:57 | sean-k-mooney | actully its not that filter | |
| 14:06:05 | sean-k-mooney | thats coming form https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L490-L498 | |
| 14:06:38 | sean-k-mooney | so it means that the host has been added to the ignored host list | |
| 14:06:52 | sean-k-mooney | what release of openstack are you using by the way | |
| 14:06:56 | ozzzo | Train | |
| 14:07:16 | ozzzo | working on upgrading from centos 7-8 to prepare for train-ussuri | |
| 14:08:41 | sean-k-mooney | ok so you need to use debug logs in the schudler to see why the hsot got filtered out | |
| 14:08:51 | ozzzo | ok | |
| 14:09:20 | sean-k-mooney | something added the host to the ignored list becaue it failed a filter | |
| 14:21:32 | sean-k-mooney | gibi: im +1 on your pps spec but i have two things i woudl like to highlihgt quickly if you have time | |
| 14:22:04 | sean-k-mooney | https://review.opendev.org/c/openstack/nova-specs/+/785014/8/specs/xena/approved/qos-minimum-guaranteed-packet-rate.rst#328 and https://review.opendev.org/c/openstack/nova-specs/+/785014/8/specs/xena/approved/qos-minimum-guaranteed-packet-rate.rst#427 | |
| 14:22:41 | sean-k-mooney | gibi: neither are blocker to me i just want to make sure we are on the same page | |
| 14:23:44 | gibi | sean-k-mooney: sure | |
| 14:23:55 | gibi | sean-k-mooney: reading... | |
| 14:41:21 | ozzzo | sean-k-mooney: Here's the debug log: https://paste.ubuntu.com/p/4MrhjJs4r6/ | |
| 14:41:35 | ozzzo | at line 102 it says "Host filter ignoring hosts: chrnc-area51-os-compute-02.chtrse.com, chrnc-area51-os-compute-01.chtrse.com" | |
| 14:41:41 | ozzzo | but it's not obvious why | |
| 14:42:16 | ozzzo | compute-01 is the one where I did the grub trick | |
| 15:32:51 | stephenfin | sean-k-mooney: looks like that stable l-c patch for os-vif failed. I assume you're going to look at disabling those now, or should I do it? | |
| 15:33:52 | ozzzo | sean-k-mooney: turns out it's in the conductor log: https://paste.ubuntu.com/p/tNYS3jzXsT/ | |
| 15:34:24 | ozzzo | at 221 it says "nova.exception.MigrationPreCheckError: Migration pre-check error: CPU doesn't have compatibility." | |
| 15:34:33 | ozzzo | and then "internal error: Unknown CPU feature arch-facilities" | |
| 15:36:16 | ozzzo | actually it looks like the problem starts at line 35: | |
| 15:36:18 | ozzzo | 2021-05-26 14:30:35.457 23 DEBUG nova.conductor.tasks.live_migrate [req-d5ac7239-4326-4d9e-9731-e4fe53a19929 - - - - -] Skipping host: chrnc-area51-os-compute-02.chtrse.com because: Migration pre-check error: CPU doesn't have compatibility. | |
| 15:37:48 | sean-k-mooney | oh the arch-facilities thing | |
| 15:38:12 | openstack | bugzilla.redhat.com bug 1658406 in libvirt "mode="host-model" VMs include broken "arch-facilities" flag name [libvirt]" [Urgent,Closed: errata] - Assigned to jdenemar | |
| 15:38:12 | ozzzo | when I google it I find this: https://bugzilla.redhat.com/show_bug.cgi?id=1658406 | |
| 15:38:25 | sean-k-mooney | thats a different upgrade issue where they chagne that between centos 8 and centos 7 | |
| 15:45:56 | ozzzo | sean-k-mooney: how can I work around it? | |
| 15:46:20 | sean-k-mooney | stephenfin: we have two fixes that work | |
| 15:47:12 | sean-k-mooney | stephenfin: https://review.opendev.org/c/openstack/os-vif/+/792840 fixed lc and https://review.opendev.org/c/openstack/os-vif/+/793155 also fix it | |
| 15:47:38 | sean-k-mooney | zuul faild for other reasons | |
| 15:49:14 | sean-k-mooney | ozzzo: i think you need to update the version fo qemu then hard reboot the guest | |
| 15:49:34 | sean-k-mooney | ozzzo: i dont think there is a fix for that that does not invovle a guest reboot | |
| 15:50:25 | sean-k-mooney | ozzzo: so effectivly you cant avoid guest downtime so your simplest solution would be to cold migrate | |
| 15:51:02 | ozzzo | ok I'll experiment with cold-migration | |
| 15:51:11 | sean-k-mooney | ozzzo: unfortnetly this is a libvirt abi breakage that we cannot fix in nova | |
| 15:52:51 | ozzzo | sean-k-mooney: got it. thanks for your advice! | |
| 15:53:55 | gibi | sean-k-mooney: replied in https://review.opendev.org/c/openstack/nova-specs/+/785014 I think we are more or less on the same page | |
| 15:55:46 | sean-k-mooney | gibi: cool ill quickly skim over them one sec | |
| 15:58:07 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Repropose "vm-boot-with-unaddressed-port" spec https://review.opendev.org/c/openstack/nova-specs/+/793199 | |
| 16:02:40 | sean-k-mooney | gibi: cool i agree with youre respeonces so +w but just want to reinfoce that there is no way currently to make a neutron extention required unless nova is willign to stop supportin all neutron backends | |
| 16:03:17 | sean-k-mooney | gibi: i have brought up this problem at the last couple of ptgs | |
| 16:03:22 | gibi | sean-k-mooney: ack, then it means we will keep support for both format for a while | |
| 16:03:36 | gibi | and I will look into adding support for making someting mandatory in neutron | |
| 16:03:49 | sean-k-mooney | gibi: yes but if more them me are asking for this maybe we can actully get neutron to add a way | |