| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-12-21 | |||
| 12:24:43 | sean-k-mooney | ya https://zuul.opendev.org/t/openstack/build/2874580c83d0484593f800d6d9c6fde2/log/controller/logs/screen-keystone.txt#89780 os the connection form keyston to the db was killed im guessign the db was killed | |
| 12:25:03 | sean-k-mooney | yep https://zuul.opendev.org/t/openstack/build/2874580c83d0484593f800d6d9c6fde2/log/controller/logs/syslog.txt#7394 | |
| 12:25:21 | sean-k-mooney | dvo-plv: ya unfortunetly | |
| 12:25:30 | sean-k-mooney | we try to optimise the jobs so that it wont happen | |
| 12:25:41 | sean-k-mooney | but sometimes it will still cause a failure | |
| 12:26:22 | sean-k-mooney | with that siad i need to fix the spellign mistaks in my devstack patch to improve this situration | |
| 12:26:52 | dvo-plv | I see, thank you | |
| 12:27:27 | sean-k-mooney | im hoping https://review.opendev.org/c/openstack/devstack/+/890693 will fix it or at least help | |
| 12:27:39 | sean-k-mooney | since you remineded me of that ill go fix the spelling issue now | |
| 12:28:05 | dvo-plv | you are welcome) | |
| 13:52:29 | opendevreview | alisafari proposed openstack/nova stable/2023.1: Fix traits to cpu flags mapping https://review.opendev.org/c/openstack/nova/+/903444 | |
| 13:56:57 | opendevreview | Sylvain Bauza proposed openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 13:56:58 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP returns mdevs https://review.opendev.org/c/openstack/nova/+/904209 | |
| 13:56:58 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 15:25:27 | opendevreview | Sylvain Bauza proposed openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 15:25:28 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP returns mdevs https://review.opendev.org/c/openstack/nova/+/904209 | |
| 15:25:28 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 16:19:15 | opendevreview | Alex Welsh proposed openstack/nova master: Add --update flag for map_cell0 https://review.opendev.org/c/openstack/nova/+/903140 | |
| 16:44:04 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.2: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/904225 | |
| 16:44:05 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.2: Allow best effort sending of notifications https://review.opendev.org/c/openstack/nova/+/904226 | |
| 16:45:19 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.1: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/904227 | |
| 16:45:20 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.1: Allow best effort sending of notifications https://review.opendev.org/c/openstack/nova/+/904228 | |
| 16:47:15 | opendevreview | Artom Lifshitz proposed openstack/nova stable/zed: Call Neutron immediately upon _post_live_migration() start https://review.opendev.org/c/openstack/nova/+/904229 | |
| 16:47:16 | opendevreview | Artom Lifshitz proposed openstack/nova stable/zed: Allow best effort sending of notifications https://review.opendev.org/c/openstack/nova/+/904230 | |
| 16:54:14 | gksk | Hi Team | |
| 16:54:22 | gksk | Did any one face this error | |
| 16:54:24 | gksk | ERROR oslo.messaging._drivers.impl_rabbit [-] [2940a4d9-ce1c-4d6c-a51d-771c7c2e16a0] AMQP server on x.x.x.x is unreachable: Server unexpectedly closed connection. | |
| 16:54:51 | gksk | I am getting this error on nova-api log when I try to spin a vm on specific compute node | |
| 16:54:55 | sean-k-mooney | that generally just means that either the tcp connection dropped or the rabbitmq server was restarted | |
| 16:55:03 | sean-k-mooney | oh ok | |
| 16:55:13 | sean-k-mooney | that can be ignored entirly | |
| 16:55:43 | sean-k-mooney | well it depends so is it causing an error with booting the vm | |
| 16:56:13 | gksk | any solution for this issue? | |
| 16:56:37 | sean-k-mooney | gksk: ther eare two reason this happens oen is not an error the other means you had an infra probelm | |
| 16:56:53 | sean-k-mooney | are you runign nova api under a wsgi server like appche | |
| 16:57:25 | sean-k-mooney | if you are and its not causing the vm creation to fail | |
| 16:57:46 | gksk | No basically when I try to create another vm on the same host (compute node ) immediately the previous vm is coming to active state | |
| 16:58:01 | sean-k-mooney | then this is just because apache kills the wsgi application after the request suspendign the heatbeat and causing the connection to close | |
| 16:58:23 | gksk | do we have any fix for this | |
| 16:58:26 | gksk | ? | |
| 16:59:04 | sean-k-mooney | it depend on why the conenction was closed | |
| 16:59:23 | sean-k-mooney | we have once ase where this is prented that we do not consider a bug as it not fixable in nova | |
| 16:59:38 | sean-k-mooney | if it impacts the spawnign fo a vm and result in it going to error | |
| 16:59:51 | gksk | I am not sure why conenction was closed | |
| 16:59:51 | sean-k-mooney | it might be a diffent edgecase that might be a fixable bug | |
| 17:00:31 | sean-k-mooney | lets start by determing how your runing nova-api | |
| 17:00:43 | sean-k-mooney | are you using the nova-api binary or runing it as a wsgi applcition on a webserver | |
| 17:01:30 | gksk | no I am using nova-api binary | |
| 17:02:06 | sean-k-mooney | ok in that case that messge would indicate that the tcp connection to rabbit was closed likely due to a networkign issue | |
| 17:02:25 | sean-k-mooney | when using nova-api directly it runnign under the eventlet webserver | |
| 17:03:06 | sean-k-mooney | that webserver will not terminate the rabbitmq connect based on the time form the last request | |
| 17:03:18 | sean-k-mooney | it will keep it open wiht a heatbeat | |
| 17:03:42 | sean-k-mooney | can you confrim taht you have https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.heartbeat_in_pthread set ot false | |
| 17:04:24 | gksk | Let me check | |
| 17:04:28 | gksk | give me 2 mins | |
| 17:06:31 | sean-k-mooney | heartbeat_in_pthread=true is only for runing under a wsgi server like apache mod_wsgi and is knwo to break running under eventlet | |
| 17:06:47 | sean-k-mooney | so in your config its imporant to set that to false | |
| 17:06:53 | sean-k-mooney | which is the default | |
| 17:07:22 | opendevreview | Sylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 17:07:23 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 17:09:30 | gksk | Hi Sean | |
| 17:09:40 | gksk | heartbeat_in_pthread is false in nova conf | |
| 17:09:46 | gksk | but its commented out | |
| 17:09:46 | sean-k-mooney | ack | |
| 17:09:53 | sean-k-mooney | ya thats the default | |
| 17:09:57 | gksk | do we need to uncomment it | |
| 17:09:58 | sean-k-mooney | so that is fine | |
| 17:10:09 | sean-k-mooney | no its false unless you set it otherwise | |
| 17:10:16 | gksk | okay | |
| 20:02:08 | opendevreview | Merged openstack/nova stable/2023.1: Fix traits to cpu flags mapping https://review.opendev.org/c/openstack/nova/+/903444 | |
| #openstack-nova - 2023-12-22 | |||
| 02:02:31 | opendevreview | melanie witt proposed openstack/nova master: block_device: Add encryption attributes to swap disks https://review.opendev.org/c/openstack/nova/+/884312 | |
| 02:02:32 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 02:02:32 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 02:02:33 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 02:02:33 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 02:02:34 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 02:02:34 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870933 | |
| 02:02:35 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 02:02:36 | opendevreview | melanie witt proposed openstack/nova master: Support rebuild and unshelve with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870939 | |
| 02:02:36 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870937 | |
| 02:02:38 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 02:02:38 | opendevreview | melanie witt proposed openstack/nova master: Support rescue with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/873675 | |
| 02:02:40 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 02:02:40 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313 | |
| 02:02:42 | opendevreview | melanie witt proposed openstack/nova master: Reject resize API requests with conflicting ephemeral encryption https://review.opendev.org/c/openstack/nova/+/904240 | |
| 06:31:03 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 06:31:03 | opendevreview | melanie witt proposed openstack/nova master: Reject resize API requests with conflicting ephemeral encryption https://review.opendev.org/c/openstack/nova/+/904240 | |
| 06:31:04 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 06:31:04 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313 | |
| 06:42:25 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| 07:00:43 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 10:46:08 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 10:46:08 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 10:46:57 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 10:48:07 | bauzas | aaaand I'm done with writing the mdev live-migration series \o/ | |
| 10:48:30 | bauzas | (well, I need to update the tests for sure, but people can start to look at the series) | |
| 13:52:46 | sean-k-mooney | bauzas: it would be interestign to see what happens if you treid that with mtty | |
| 13:53:09 | bauzas | sean-k-mooney: I'll do this when I'm back from pto | |
| 13:53:30 | sean-k-mooney | ack even if it technially does not supprot it yet im wondering if qemu/libvirt will allow it | |
| 13:53:52 | bauzas | you mean about mtty ? | |