| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-23 | |||
| 13:39:14 | sean-k-mooney | and it just does not pic those up via dhcp? | |
| 13:39:16 | EugenMayer | yes i do | |
| 13:39:29 | EugenMayer | it ignores it on ubuntu, exactly | |
| 13:39:39 | sean-k-mooney | ya that sound like a dhcpclint bug in ubuntu 20.04 | |
| 13:40:07 | sean-k-mooney | i assume they are both useing the came dhcp clinet implemenation by default | |
| 13:40:13 | EugenMayer | yes debian works. Same network same everything (using terraform here, so just replaced the image) | |
| 13:41:00 | sean-k-mooney | perhap its fixed in a later cloud image? im not sure if you are using the latest point release image of 20.04 but might be worth trying the nightly image | |
| 13:41:09 | EugenMayer | using the latest one | |
| 13:41:13 | EugenMayer | not older then 5 days | |
| 13:41:24 | EugenMayer | did you find a bug reeport? | |
| 13:41:37 | sean-k-mooney | no have not look | |
| 13:42:03 | EugenMayer | i look it up and try > 20.xx | |
| 13:42:10 | sean-k-mooney | it used to work i think but that the only thing i can think of is a bug in the dhcpclient or the cloud-init default behavior in ubuntu | |
| 13:42:26 | EugenMayer | maybe it is netplan related | |
| 13:43:37 | EugenMayer | i use what is referenced at Same does me. Spinnging up a debian box right now, expecting it to work about | |
| 13:43:40 | EugenMayer | sorry | |
| 13:43:45 | EugenMayer | here https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/configure-openstack.html#set-up-public-networking | |
| 13:49:46 | EugenMayer | interesting, resolvectl status shows up the right dns serevers... but the resolution of my internal domains does not work | |
| 13:49:57 | EugenMayer | i have to use 'dig @IP domain' | |
| 13:56:30 | bauzas | lyarwood: oh, sorry, I haven't seen https://blueprints.launchpad.net/nova/+spec/ephemeral-encryption-libvirt | |
| 13:56:41 | bauzas | lyarwood: then we need to update the spec to tell that the BP is ^ | |
| 13:56:51 | bauzas | lyarwood: I mean, in https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/ephemeral-encryption-libvirt.html | |
| 13:57:32 | lyarwood | bauzas: yeah there's a fup above for that now | |
| 13:59:30 | lyarwood | https://review.opendev.org/c/openstack/nova-specs/+/818917 sorry | |
| 14:03:15 | EugenMayer | sean-k-mooney ok i found it. Damn i confused the isolated test. I had a typo in my CNAME thus the isolated test with a blank 20.04 box failed and i assumed a general issues. Now vanilla focal works. just focal + rke2 has the wrong DNS servers, which might be an RKE2 setup thing. Sorry for wasting time, got dragged away, missed that typo | |
| 14:04:01 | sean-k-mooney | EugenMayer: no worries | |
| 14:19:40 | opendevreview | Merged openstack/nova-specs master: fup: Correct the libvirt ephemeral encryption blueprint link https://review.opendev.org/c/openstack/nova-specs/+/818917 | |
| 14:33:11 | bauzas | eeeeek https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d | |
| 14:33:17 | bauzas | sean-k-mooney ^ | |
| 14:34:13 | mnaser | gibi: yeah i saw your comments and found that issue, it seems like even with retry=0, it still is kinda failing | |
| 14:34:20 | mnaser | since it gets stuck in a reconnecting loop | |
| 14:34:27 | bauzas | sean-k-mooney: context is https://bugs.launchpad.net/nova/+bug/1951656 | |
| 14:34:51 | gibi | mnaser: yepp the retry config is only used to message sending but not connection setup | |
| 14:35:16 | mnaser | that's where i ended up and it felt pretty non-trivial at that point to go from there :( | |
| 14:35:34 | mnaser | at least, it's beyond my scope of comfortable knowledge anyways | |
| 14:35:38 | gibi | mnaser: it seems the current oslo.messaging driver interface does not have way to express that the consumer wants configurable retry for conenction setup | |
| 14:36:03 | sean-k-mooney | bauzas: we proably shoudl jsut not use libvirt for mdevs at all honestly but fun | |
| 14:36:09 | gibi | mnaser: I'm still digging oslo.messaging to understand more how to fit this in | |
| 14:36:32 | bauzas | sean-k-mooney: well, I'd prefer the other way, ie. asking libvirt to create mdevs | |
| 14:36:41 | mnaser | gibi: something i found interesting was - https://docs.celeryproject.org/projects/kombu/en/stable/_modules/kombu/connection.html#Connection.ensure_connection | |
| 14:36:42 | bauzas | sean-k-mooney: but they won't do it | |
| 14:36:58 | mnaser | it looks like kombu has it's own error handling inside ensure_connection that maybe didn't exist back when openstack started using it | |
| 14:37:11 | sean-k-mooney | bauzas: well the reason i say that is libvirt does not define that as a stable interface | |
| 14:37:29 | sean-k-mooney | and createing/remvoing mdevs via the filesystem is pretty trivial | |
| 14:37:35 | gibi | mnaser: yeah komubo has the flags in the interface | |
| 14:37:40 | gibi | kombu | |
| 14:37:54 | bauzas | anyway, when I'm seeing it, I wonder whether libvirt folks know there are some upper services that use them | |
| 14:37:59 | gibi | mnaser: so we could configure kombu to stop after x retries | |
| 14:38:12 | gibi | mnaser: we just don't have the scaffolding in oslo.messaging to use that | |
| 14:38:23 | mnaser | gibi: yeah but i think then that brings the other interesting issue of 'does that mean it gives up forever for all future notifications' | |
| 14:38:24 | sean-k-mooney | bauzas: they are partly aware i had a converstaion with them about this in relateino to vdpa | |
| 14:38:42 | sean-k-mooney | and mac adresses | |
| 14:38:58 | sean-k-mooney | they do not guarenttee the names are stabel and said we shoudl avoid realying on them | |
| 14:40:11 | bauzas | meh | |
| 14:40:13 | bauzas | OK | |
| 14:40:16 | sean-k-mooney | bauzas: to me this is incorrect behavior in mdevctl and the libvirt chagne should be reverted | |
| 14:40:22 | bauzas | gibi: heh https://bugs.launchpad.net/nova/+bug/1951623 | |
| 14:40:36 | bauzas | gibi: I guess this will be fixed by your change, right ? | |
| 14:41:40 | bauzas | oh, nevermind, it was for reboot | |
| 14:43:52 | mnaser | i do see a point in 'if notifications are down, stop doing things that can be billed' | |
| 14:45:10 | gibi | bauzas: unfortunatly that is a different problem | |
| 14:45:16 | sean-k-mooney | mnaser: that is anyting other then delete including allowing the instance to continue running :) | |
| 14:45:26 | bauzas | gibi: looks like, yes | |
| 14:45:39 | mnaser | sean-k-mooney: but that means more $$$ am i rite? =ap | |
| 14:46:09 | gibi | mnaser: yepp, I agree, but I think this behavior needs to be configurable | |
| 14:46:40 | gibi | mnaser: to handle the case when notification only just good to have | |
| 14:46:43 | gibi | but not must have | |
| 14:47:06 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Explode libvirt builds to trigger tempest error code https://review.opendev.org/c/openstack/nova/+/818932 | |
| 14:47:16 | sean-k-mooney | mnaser: :) i think the correct behavior if notification cant be sent is either to just log an error and drop them or continue to have the service work but set its healtch check to degraded | |
| 14:47:34 | sean-k-mooney | the only other option i see would be for the service to terminate | |
| 14:48:01 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Explode libvirt builds to trigger tempest error code https://review.opendev.org/c/openstack/nova/+/818932 | |
| 14:48:20 | sean-k-mooney | mnaser: if we are going to stop some operation from workign i dont think we shoudl be reportign the srvice as up in our current status filed | |
| 14:48:37 | sean-k-mooney | but that is based on a heatbeat which might still be working | |
| 14:48:44 | sean-k-mooney | so that not really something we can change | |
| 14:49:40 | mnaser | sean-k-mooney: historically i rather a failure and consistent behaviour than a degraded 'pass' that leaves a bit of a mess | |
| 14:49:45 | mnaser | (see: ignore cinder volume deletes) | |
| 14:49:59 | sean-k-mooney | well notifocation are generally considerd besteffort | |
| 14:50:09 | sean-k-mooney | its not a mission cirtical part of the cloud | |
| 14:50:57 | sean-k-mooney | so we could make the service exit as an opt in behavior but in generally i think logging an error and reportign a degrated health status then allowing something external to decied what to do i think is better | |
| 14:51:15 | mnaser | i guess notifications being critical will dpeend on who you ask ;p | |
| 14:51:45 | sean-k-mooney | form a nova point of view i dont think they ever have had any guarentees | |
| 14:52:32 | sean-k-mooney | i know people do consume them for billing and other uses but the in ablity to send a notificaotn shoudl not have any impact on a user operation by default | |
| 14:53:59 | sean-k-mooney | mnaser: as a customer if i tried to reboot a vm and it failed because fo a notrifcaiton issue i wold expect a 500 or other error code to be logged showing its an infra issue so i can file a ticket with my cloud provider for downtime | |
| 14:54:34 | mnaser | sean-k-mooney: that's true, but i guess since we don't have any buffering right now, it means notifications can be lost .. and in theory, they'll get a 500 anyways now if the notifications are down =P | |
| 14:54:53 | sean-k-mooney | mnaser:basicly in cloud where notificaiotn are treated as critical it would be a cloud outage from a customer point of ivew | |
| 14:55:17 | sean-k-mooney | mnaser: hehe i guess they will get the 500 now ture | |
| 14:55:32 | sean-k-mooney | although only if you dont set a retry limit right | |
| 14:55:40 | mnaser | nope, even if you set retry=0 it fails :( | |
| 14:55:51 | sean-k-mooney | oh ok then that is a bug | |
| 14:55:51 | mnaser | see what gibi mentioned above, it gets stuck on trying to connect and that has no retries | |
| 14:55:56 | mnaser | it loops forever on trying to connect | |
| 14:56:04 | sean-k-mooney | ah right | |
| 14:56:27 | sean-k-mooney | because its the amqp connection that is down its not an issue with sendign to a queue or exchange | |
| 14:56:32 | mnaser | i captured GMR and updated info here - https://bugs.launchpad.net/nova/+bug/1917645 | |
| 14:57:00 | sean-k-mooney | mnaser: well that proably should be converted to an oslo messaging bug at this point | |
| 14:57:22 | mnaser | sean-k-mooney: good point actually | |
| 14:57:47 | sean-k-mooney | nova might be able to work around this but eventually we will want to adress this there i suspect | |
| 15:01:11 | opendevreview | Dmitrii Shcherbakov proposed openstack/os-traits master: Add a trait for remote_managed port-capable nodes https://review.opendev.org/c/openstack/os-traits/+/818514 | |
| 15:03:42 | gibi | sean-k-mooney: sure oslo.messaging lacks a configuration possibility to stop retrying the connect, but I don't think it is purelya bug | |
| 15:03:58 | sean-k-mooney | pro tip dont put a folder with 2412 sub directories in your test path if you want test discovery to happen before the heat death of the universe | |