| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-25 | |||
| 14:23:04 | sean-k-mooney | we worked aroudn it with https://github.com/lyarwood/cirros/commit/c8972ca3dcde2bedc00cbeda0e0ccb8fcbf92883 but chattign to lee i found a fixed kernel bug that had been released in an newer verison | |
| 14:23:18 | kashyap | Oh, this, the IO-APIC one. | |
| 14:23:47 | sean-k-mooney | ya so there was a fix for the apic issue in a later kernel package | |
| 14:23:56 | sean-k-mooney | we worked around it with the kernel parmater | |
| 14:24:09 | sean-k-mooney | but the package never got updated at least not at the time | |
| 14:24:17 | sean-k-mooney | it may have been since | |
| 14:24:43 | kashyap | Ah, nod | |
| 15:35:25 | dansmith | kashyap: from this morning: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3e/883602/2/gate/nova-ceph-multistore/f3e5585/testr_results.html | |
| 15:35:35 | dansmith | guest kernel crash, but it's the IO-APIC variant | |
| 15:36:11 | kashyap | dansmith: Ohh, is it with CirrOS 6.1? | |
| 15:36:25 | dansmith | should be, it's from this morning on master | |
| 15:36:57 | kashyap | If it's IO-APIC variant one; then the immediate thing that comes to mind is does this image also needs "no_timer_check" to its kernel parameter | |
| 15:37:01 | dansmith | yep https://zuul.opendev.org/t/openstack/build/f3e558513e3f4a6c8e9c0ceb50a023e1/log/job-output.txt#2056 | |
| 15:37:29 | dansmith | well, it ran a hundred other tests with that image and configuration of course | |
| 15:37:42 | dansmith | but if that's something we should always be using in devstack, you should add that :) | |
| 15:39:19 | kashyap | Understood; I've added it to my plate to see if there's any other way around here, besides the kernel parameter | |
| 15:40:53 | frickler | kashyap: I missed that kernel bump issue because I was on holiday. iiuc it will still need a release, like 0.5.3? | |
| 15:41:12 | kashyap | dansmith: Why do you mention the hundred other tests? Is it as a "contrasting" success of what tests are actually passing? | |
| 15:41:36 | dansmith | kashyap: no I'm saying it's clearly not "this requires this flag to work at all" | |
| 15:41:38 | kashyap | frickler: See the issue mentioned here: https://github.com/cirros-dev/cirros/issues/102 | |
| 15:41:59 | frickler | kashyap: yes, I don't see a release for that | |
| 15:42:40 | frickler | I'l check with smoser | |
| 15:42:59 | kashyap | dansmith: Aah, like that. Noted. I was only thinking of th IO-APIC + timer panic | |
| 15:43:25 | kashyap | frickler: smoser seems to point me to a build? | |
| 15:43:43 | kashyap | frickler: Please mention your thoughts on that ticket, to keep everything in one place | |
| 15:45:58 | frickler | yes, that's kind of a testing build. updated the issue, good point | |
| 19:53:09 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Translate VF network capabilities to port binding https://review.opendev.org/c/openstack/nova/+/884439 | |
| 22:19:13 | opendevreview | melanie witt proposed openstack/nova master: WIP raw imagebackend support https://review.opendev.org/c/openstack/nova/+/884313 | |
| 22:20:34 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| #openstack-nova - 2023-05-26 | |||
| 00:25:19 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 00:25:19 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 00:25:20 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 00:25:20 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 00:25:21 | opendevreview | melanie witt proposed openstack/nova master: Support rescue with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/873675 | |
| 00:25:21 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 00:25:22 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 00:25:22 | opendevreview | melanie witt proposed openstack/nova master: WIP raw imagebackend support https://review.opendev.org/c/openstack/nova/+/884313 | |
| 00:32:25 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 00:43:39 | opendevreview | Wenping Song proposed openstack/nova stable/victoria: Trival Change: Remove redundant code in instance delete https://review.opendev.org/c/openstack/nova/+/796912 | |
| 02:40:09 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 14:37:30 | lowercase | sean-k-mooney: It appears we are hitting a bug where instances are not honoring availability zone placement even though we have cinder_Cross_az attach set to false. Is this ringing any bells or anything? | |
| 14:37:36 | lowercase | Xena release | |
| 15:01:41 | bauzas | lowercase: are you sure that the instances are not pinned ? | |
| 15:05:25 | lowercase | bauzas: yes, we don't do cpu pinning. | |
| 15:06:10 | bauzas | lowercase: no sorry, I mean, pinned to an AZ | |
| 15:07:05 | lowercase | how would we pin them to an AZ? | |
| 15:08:34 | lowercase | so what is happening is that a when we attempt to resize an instance, the nova scheduler can (in our experience) pick a host outside of the AZ where the instance is currently is, because we have cinder_cross_az attach set to false | |
| 15:08:47 | lowercase | because we have multiple ceph clusters, 1 ceph cluster per AZ | |
| 15:13:31 | lowercase | https://docs.openstack.org/nova/xena/admin/availability-zones.html#implications-for-moving-servers | |
| 15:13:38 | lowercase | #4 is our condition | |
| 15:30:58 | sean-k-mooney | lowercase: the schgduler is allowed ot pick an instance outside the current az | |
| 15:31:14 | sean-k-mooney | lowercase: it is only not allowe to do that if the instnace is create reqeusting a specific AZ | |
| 15:31:55 | lowercase | is there anyway I can retro-give an instance the paramter to keep the instance in a az | |
| 15:32:08 | sean-k-mooney | lowercase: implications-for-moving-servers is refering to moveing compute hosts not vms | |
| 15:32:29 | lowercase | interesting | |
| 15:32:52 | lowercase | the only way to pin a vm to an AZ is ONLY at creation time? | |
| 15:32:56 | sean-k-mooney | retoactivly on xena | |
| 15:33:29 | sean-k-mooney | i think vai shelve and then unshelve specifyign an az | |
| 15:33:58 | lowercase | hmmm.. that's not a bad idea actually. | |
| 15:34:36 | lowercase | is it possible to retroactively pin a vm to az by modifying the database? | |
| 15:34:58 | sean-k-mooney | yes... | |
| 15:35:11 | sean-k-mooney | you can set the az in teh request spec which is a json blob | |
| 15:35:27 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id70 | |
| 15:35:39 | sean-k-mooney | unshlve to an az was added in train just an fyi | |
| 15:36:46 | sean-k-mooney | lowercase: https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L75 is what determins what the schduler requests | |
| 15:37:16 | sean-k-mooney | so you can just set the az in the json blob in the api db | |
| 15:37:20 | sean-k-mooney | but its error prone | |
| 15:37:24 | lowercase | shelving and unshelving requires to shutdown the instance. We would prefer a diff method that retroactively restrict vms to an az. | |
| 15:37:52 | lowercase | that's why I could script it out to alter the database would be preferable | |
| 15:38:38 | sean-k-mooney | i feel like this is one of those things that comes in phases. | |
| 15:38:53 | sean-k-mooney | we have had 2 or 3 diffent people ask about this in the last month | |
| 15:39:31 | sean-k-mooney | a nova-manage command to update the requested az in the request spec might be useful | |
| 15:39:46 | sean-k-mooney | i.e. one that woudl set it to whatever the instance currently has | |
| 15:40:21 | lowercase | We just entered a new market, where our openstck environment is still in flux. Typically, when vms are built they don't shutdown or get moved (or rarely). Since we have a new requirements in this new location, the customers vms have been in flux more than normal for us. | |
| 15:40:28 | sean-k-mooney | something like "nova-mange instance az sync <uuid>" | |
| 15:40:36 | lowercase | that would be very cool | |
| 15:41:08 | sean-k-mooney | its not nessisaly safe to set an az request that does not match the instance current value | |
| 15:41:25 | sean-k-mooney | but it shoudl always be safe to sync the current value to the request spec | |
| 15:41:43 | sean-k-mooney | well provided th instance is not in error | |
| 15:42:14 | sean-k-mooney | we can asusme that it its running in az 1 its safe to pin it to az1 going forward if you ask use too | |
| 15:48:59 | lowercase | ( we are having internal team dialogue, just a moment) | |
| 15:55:02 | lowercase | Okay, we are happy with your current suggestions. | |
| 15:55:19 | lowercase | A nova-mange instance az sync <uuid> would def be helpful in the future | |
| 15:55:40 | lowercase | I need to go afk, we are going to pick this up again after lunch. | |
| 15:55:43 | lowercase | thank you for your time. | |
| 15:55:48 | lowercase | We certainly appreciate it. | |
| 16:10:05 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882321 | |
| 16:10:06 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882322 | |
| 17:42:28 | bauzas | lowercase: sorry I forgot you :facepalm: | |
| 17:43:00 | bauzas | lowercase: sean-k-mooney: if an instance is created using a volume, then we pin the AZ by the RequestSpec | |
| 17:44:13 | bauzas | lowercase: but we have https://bugs.launchpad.net/nova/+bug/2018318 in case an enduser wants to create a 'boot-from-volume' instance by using an image that will be creating a volume | |
| 17:51:04 | lowercase | Sorry, internet went down. I receieved your :42 and :43 messages. | |
| 17:53:09 | lowercase | Tested the shelving/unshelving. It does fix the problem BUT it won't be a good solution for us | |
| 17:53:15 | lowercase | The way it works is it powers off the instance and snapshots. When you unshelve it boots the snapshot | |
| 17:53:21 | lowercase | That is very problematic because snapshots blow up qcows and it would not be practical at all with our instances | |
| 17:53:26 | lowercase | So looking at what updating the extra specs in the databse would be like | |
| 17:59:17 | lowercase | rebooting my computer, brb | |
| 17:59:26 | dansmith | thats what shelve does | |
| 18:03:32 | lowercase | back | |
| 18:05:04 | dansmith | [10:59:26] <dansmith> thats what shelve does | |