| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-25 | |||
| 17:07:34 | sean-k-mooney | oh white space errors remind me again that a -3 right :) ill review it shortly it look pretty short and too the point | |
| 17:07:50 | sean-k-mooney | actully im being summoned to go for a walk so when i get back | |
| 17:09:18 | gmann | ok, I was searching whitespace in stephenfin spec :P | |
| 17:10:12 | sean-k-mooney | hehe no in yours https://review.opendev.org/c/openstack/nova-specs/+/793011/1/specs/xena/approved/allow-project-admin-list-hypervisors.rst#39 | |
| 17:10:42 | gmann | yeah :) | |
| 17:11:02 | gmann | stephenfin: sean-k-mooney gibi bauzas I have +W this https://review.opendev.org/c/openstack/nova-specs/+/783827/2 and +2 on follow up 793010 | |
| 17:12:26 | gibi | gmann: ack on both count | |
| 17:23:31 | openstackgerrit | Merged openstack/nova-specs master: Repropose configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/783827 | |
| 17:42:34 | ozzzo | sean-k-mooney: I set cpu_models = Cascadelake-Server-noTSX,Skylake-Server-IBRS in nova.conf and also: GRUB_CMDLINE_LINUX="crashkernel=auto tsx=on" in /etc/default/grub, and rebooted, and then redeployed the compute, but I still get "ERROR oslo_service.service nova.exception.InvalidCPUInfo: Configured CPU model: Skylake-Server-IBRS is not compatible with host CPU." in nova-compute.log | |
| 18:09:45 | ozzzo | looks like I might need to use grubby; reading man page | |
| 18:17:55 | ozzzo | sean-k-mooney: that fixed it, ty! | |
| 18:34:41 | ozzzo | sean-k-mooney: that fix it in the sense that I can now start the nova-compute service with the new settings, but VMs still refuse to migrate with the same " Host filter ignoring hosts:" error | |
| 18:59:16 | ozzzo | sean-k-mooney: when I check the hypervisor now, cpu_info has changed from "Cascadelake-Server-noTSX" to "Cascadelake-Server" | |
| 18:59:27 | ozzzo | so it still isn't matching "Skylake-Server-IBRS" | |
| 19:00:15 | ozzzo | Trying "cpu_models=Skylake-Server-IBRS" in "/etc/kolla/config/nova.conf" | |
| 19:07:48 | ozzzo | that didn't make a difference. | |
| 19:08:44 | ozzzo | I looked inside the container, and I see "cpu_models = Skylake-Server-IBRS" in /etc/nova/nova.conf | |
| 19:09:15 | ozzzo | but the hypervisor show still says "Cascadelake-Server" | |
| 19:09:19 | ozzzo | what am I missing? | |
| 19:32:59 | masterpe | I had the same | |
| 19:37:05 | masterpe | at my place the commands says Cascadelake-Server-noTSX. This happend when I upgrade to qemu(1:4.2-3ubuntu6.13) or libvirt(6.0.0-0ubuntu8.8) | |
| 19:38:19 | masterpe | To solve this I restarted the instances and accepted that I can not live migrate those instances. | |
| 19:41:39 | ozzzo | that won't work for me. When I upgrade production, we have to be able to migrate instances there | |
| 19:44:21 | masterpe | ozzzo: I can Imagine, I was able to live migrate from a machine that was no upgraded to a node that is upgraded. | |
| 19:44:57 | ozzzo | masterpe: did you try the grub trick with tsx=on? | |
| 19:45:12 | masterpe | ozzzo: no I did not. | |
| 19:45:56 | ozzzo | I did that, and that allowed me to start nova_compute with "cpu_models=Skylake-Server-IBRS" but it looks like I'm still missing something | |
| 19:45:56 | masterpe | But that instance further live migraten to a second machine was not possible. | |
| 19:48:54 | masterpe | but Skylake- is older then Cascadelake | |
| 19:49:57 | masterpe | What does /usr/bin/virsh -r capabilities|grep \<model\>|head -1|cut -d '>' -f2|cut -d '<' -f1 | |
| 19:49:57 | masterpe | say ? | |
| 19:50:48 | masterpe | and libvirt and qemu version are you and also what processor do you have? | |
| 19:51:00 | ozzzo | my computes don't seem to have virsh installed | |
| 19:52:10 | ozzzo | I'm running kolla | |
| 19:52:14 | ozzzo | everything is in containers | |
| 19:52:44 | ozzzo | Skylake-Server-IBRS is how centos 7 recognized my compute CPUs | |
| 19:53:09 | ozzzo | centos 8 recognizes as "Cascadelake-Server-noTSX" | |
| 19:53:33 | ozzzo | and after the grub trick, it recogizes as "Cascadelake-Server" | |
| 19:54:09 | masterpe | I use the "Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz" as CPU. | |
| 19:55:09 | ozzzo | my computes have Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz | |
| 19:55:28 | masterpe | thanks | |
| 19:59:02 | masterpe | Is this helpfull? https://bugzilla.redhat.com/show_bug.cgi?id=1678381 ? | |
| 19:59:04 | openstack | bugzilla.redhat.com bug 1678381 in libvirt "[RFE] - libvirt support for cascade lake cpu - Fast Train" [Unspecified,Closed: errata] - Assigned to jdenemar | |
| 20:01:09 | masterpe | and or https://bugzilla.redhat.com/show_bug.cgi?id=1839926 ? | |
| 20:01:29 | ozzzo | masterpe: it looks like the first bug was 8.1 and I'm on 8.3 | |
| 20:01:32 | ozzzo | reading 2nd | |
| 20:25:18 | ozzzo | masterpe: I think the different CPU might be related to the 2nd one, but I think sean's grub trick should have worked around that | |
| 20:25:22 | ozzzo | I must be missing something | |
| 20:27:50 | masterpe | I saw on https://bugzilla.redhat.com/show_bug.cgi?id=1839926#c7 the hle and rtm set to disabled. | |
| 20:27:51 | openstack | bugzilla.redhat.com bug 1839926 in libvirt "5 more -noTSX x86 CPU models are newly introduced" [Unspecified,Closed: errata] - Assigned to jdenemar | |
| 20:36:12 | ozzzo | Do I need to set something in cpu_model_extra_flags? | |
| 20:36:35 | ozzzo | I see it mentioned here: https://docs.openstack.org/nova/train/admin/configuration/hypervisor-kvm.html | |
| 23:23:42 | openstackgerrit | Merged openstack/nova-specs master: Amend configurable-instance-hostnames to include response changes https://review.opendev.org/c/openstack/nova-specs/+/793010 | |
| #openstack-nova - 2021-05-26 | |||
| 08:12:47 | bauzas | good morning Nova | |
| 08:17:48 | lyarwood | Morning | |
| 08:18:41 | lyarwood | so what happens to this channel if I look in a mirror and say `Libera Chat` three times? | |
| 08:19:24 | lyarwood | https://twitter.com/fosdem/status/1397454352835653632 context | |
| 08:19:54 | lyarwood | gibi: stephenfin | |
| 08:20:03 | lyarwood | argh sorry can you hit https://review.opendev.org/c/openstack/nova/+/790660 again please | |
| 08:20:34 | lyarwood | * lyarwood is still getting use to nick auto-complete in the element matrix client | |
| 08:38:05 | gibi | sure' | |
| 09:13:00 | openstackgerrit | Vlad Gusev proposed openstack/nova stable/stein: libvirt: Ignore device already in the process of unplug errors https://review.opendev.org/c/openstack/nova/+/793044 | |
| 10:13:42 | sean-k-mooney[m] | lyarwood: i think i figured out the cause of https://bugs.launchpad.net/nova/+bug/1929446 by the way so hopefully we will be able to fix that relitivly quickly | |
| 10:13:43 | openstack | Launchpad bug 1929446 in OpenStack Compute (nova) "check_can_live_migrate_source taking > 60 seconds in CI" [Medium,Triaged] | |
| 10:14:04 | lyarwood | * lyarwood clicks | |
| 10:14:37 | lyarwood | that's blocking the other nova thread? | |
| 10:14:51 | sean-k-mooney[m] | https://zuul.opendev.org/t/openstack/build/bb6fd21b5d8c471a89f4f6598aa84e5d/log/controller/logs/screen-n-cpu.txt#12242 | |
| 10:15:11 | sean-k-mooney[m] | yep all logs form the agent stop until the timeout fires | |
| 10:17:27 | lyarwood | req-00377477-1efd-417a-b9c1-5db2f656c051 seems to run just fine in between | |
| 10:17:49 | lyarwood | and req-cb0aa064-b389-4cd5-aeb3-5c5cf0541b00 | |
| 10:18:24 | sean-k-mooney[m] | yep but that could just be down to the order in whic the green thread are resumed | |
| 10:19:39 | lyarwood | * lyarwood < https://matrix.org/_matrix/media/r0/download/matrix.org/iVBmpWasUBctVWPeNDxVklaG/message.txt > | |
| 10:19:42 | lyarwood | :| | |
| 10:19:54 | lyarwood | was just checking _check_shared_storage_test_file to see if that spawned anything | |
| 10:20:08 | lyarwood | and that caught my eye | |
| 10:21:02 | sean-k-mooney[m] | the exists check will yelid | |
| 10:22:02 | sean-k-mooney[m] | we do not need the polling behavior in os-vif | |
| 10:22:34 | sean-k-mooney[m] | so im going to see if i can disable it without having to recreate the connection every time we plug or unplug a port | |
| 10:25:34 | lyarwood | kk there is a call to libvirt in between the two logged lines for this request btw | |
| 10:26:12 | lyarwood | there's a lookupByUUIDString | |
| 10:26:17 | lyarwood | fetching the guest | |
| 10:28:54 | sean-k-mooney | ya that coudl be related as we have had similar blocking issue there too | |
| 11:23:25 | sean-k-mooney | stephenfin: you missing a comma in https://review.opendev.org/c/openstack/nova/+/792356/3/nova/tests/unit/virt/libvirt/test_vif.py#1076 | |
| 11:23:39 | sean-k-mooney | thats why the jobs failed. | |
| 11:25:33 | sean-k-mooney | ill update https://review.opendev.org/c/openstack/nova/+/777679 quickly for the nits and let you know when its done | |
| 11:36:09 | sean-k-mooney | gibi: i have one comment that i would like your opipion on https://review.opendev.org/c/openstack/nova-specs/+/787178/5/specs/xena/approved/cyborg-no-user-token.rst#56 | |
| 11:37:01 | sean-k-mooney | im +1 on the spec currently if other are ok with that then im also +2 i think. its a relitvly simple change and it mirror what we do for other services so i dont think this is really that contoversial. | |
| 11:44:21 | gibi | sean-k-mooney: you have a good point. I dropped my +2. Lets document the compromise in the secuirty section then I'm +2 again | |
| 11:44:49 | sean-k-mooney | ok works for me | |
| 12:06:32 | lyarwood | Would anyone care if we dropped the direct swap volume tests from tempest and instead just had the indirect volume migration/retype tests to exercise it? | |
| 12:06:43 | lyarwood | it's failing again at the moment on master | |
| 12:11:21 | gibi | lyarwood: is it so that the direct test is someting that a user would never do as swap volume is for cinder to call? | |
| 12:11:51 | lyarwood | admins can use it at the moment but I've wanted to move it under the external events API | |
| 12:11:54 | stephenfin | lyarwood: I don't really understand the question, so I'll defer to your judgement 0:) | |
| 12:11:56 | lyarwood | happy to wait until that's done to remove it | |
| 12:12:09 | lyarwood | lol | |
| 12:12:28 | lyarwood | anyway the CI failures a valid so I'll fix these first | |
| 12:12:53 | lyarwood | http://paste.openstack.org/show/805745/ - I assume something has changed with the way in which libvirt looks up devices | |
| 12:18:43 | gibi | lyarwood: if you have a clue how to fix it then it is better to fix first and then decide on if we want to move | |
| 12:18:55 | gibi | does the indirect test uses the API differently that it is not affected? | |