| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-07 | |||
| 00:30:22 | sean-k-mooney | oh it is | |
| 00:30:42 | sean-k-mooney | it was the compute queue on the /nova vhost | |
| 00:30:57 | sean-k-mooney | the /nova vhost is the exchange name right? | |
| 00:31:05 | mnaser | let me check | |
| 00:31:58 | mnaser | seeing a bunch of nova.*, amq.*, reply_* and compute_* one | |
| 00:32:03 | mnaser | i wonder if the compute_fanout was the missing one | |
| 00:33:26 | sean-k-mooney | in theory if we we declared the fanout consuemr using https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L1086 | |
| 00:33:44 | sean-k-mooney | when we created the connection it should be in our consumeer list and we shoudl hav erecreated it | |
| 00:34:15 | melwitt | I added a comment to the LP bug | |
| 00:34:20 | sean-k-mooney | but if we dont make our selves a consumer of a queue then we wont recarte it | |
| 00:34:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299 | |
| 00:34:47 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344 | |
| 00:35:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673 | |
| 00:35:22 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.opendev.org/576676 | |
| 00:36:19 | mnaser | sean-k-mooney: but then if restarting recreates things, why dont the existing computes start working | |
| 00:36:37 | mnaser | ooooooh hold on | |
| 00:36:51 | mnaser | compute.<hostname> queue only exists for the one that i restarted. | |
| 00:37:24 | mnaser | so when it recovers, its not declaring that again. nb: i don't know much about rabbitmq | |
| 00:37:56 | sean-k-mooney | mnaser: yes its not the fanout queue its the topic queue for that host | |
| 00:39:10 | sean-k-mooney | e.g. the one that is used for calls | |
| 00:39:58 | sean-k-mooney | mnaser: you could enable Durable queues on the rabbit side | |
| 00:40:02 | mnaser | yeah.. i'm trying to find where that all gets declared | |
| 00:40:03 | sean-k-mooney | we might have a config option for that | |
| 00:41:13 | sean-k-mooney | in oslo its one of these two fucntion https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L1051-L1084 | |
| 00:42:04 | sean-k-mooney | setting https://docs.openstack.org/nova/latest/configuration/config.html#oslo-messaging-rabbit will likely help | |
| 00:42:39 | mnaser | yeah i guess, i just kinda wanna get to the bottom of it too | |
| 00:43:06 | mnaser | openstack/nova doesnt seem to define any declare_topic_consumer | |
| 00:43:10 | sean-k-mooney | well it will help by masking the issue so the queue persist across rabbit reboots but ya we should fix it | |
| 00:46:30 | mnaser | imstruggling to find what creates those queues in nova | |
| 00:47:03 | sean-k-mooney | i might be this https://github.com/openstack/nova/blob/master/nova/rpc.py#L66-L92 | |
| 00:47:05 | mnaser | no hits on declare_topic_consumer | |
| 00:51:13 | sean-k-mooney | actully i think its via this https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/service.py#L184-L192 | |
| 00:51:25 | mnaser | https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/amqpdriver.py#L650-L665 | |
| 00:52:25 | sean-k-mooney | yes so https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/service.py#L186 | |
| 00:52:52 | sean-k-mooney | is where in the base nova serivce calse we declar the target for an rpc to the service based on the topic and host | |
| 00:54:04 | mnaser | i think that is not being recreated.. | |
| 00:54:59 | openstackgerrit | Daniel Speichert proposed openstack/nova-specs master: Directly download and upload images to RBD https://review.opendev.org/658903 | |
| 00:55:09 | sean-k-mooney | i think so too | |
| 00:55:28 | sean-k-mooney | it intally gets created here https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/cmd/compute.py#L56-L58 | |
| 00:55:52 | sean-k-mooney | when we start the agent | |
| 00:56:00 | mnaser | and i think that ends up all the way down here when it starts listneing https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/amqpdriver.py#L650-L665 | |
| 00:56:15 | mnaser | but declare_topic_consumer doesnt store state or anything, it just does it one time | |
| 00:56:29 | mnaser | so if the queue disappears, declare_topic_consumer technically never runs | |
| 00:57:35 | sean-k-mooney | right so we need to add it to our consumers | |
| 00:58:37 | sean-k-mooney | that said | |
| 00:58:39 | sean-k-mooney | https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L946-L969 | |
| 00:58:55 | sean-k-mooney | i think that shoudl be happening already | |
| 01:00:20 | sean-k-mooney | so set_current_channel shoudl redeclar it here https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L792-L793 | |
| 01:01:22 | sapd1 | Hi everyone, I would like to regenerate metadata/properties in libvirt xml (I mean virsh dumpxml output) with running instances. How can I do that without reboot instances :D | |
| 01:01:29 | sean-k-mooney | we do not call self._set_current_channel here however https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L756 | |
| 01:02:01 | sean-k-mooney | sapd1: you cant really what are you actully trying to do | |
| 01:02:02 | mnaser | i didn't hit that ecxeption i think | |
| 01:03:16 | sean-k-mooney | sapd1: i mean you can update the xml with virsh edit but im not sure why you want to modify the metadata in the xml | |
| 01:03:48 | sean-k-mooney | mnaser: ya if you are hitting the generic exception below that you shoudl end up redeclarign the queue | |
| 01:04:12 | sean-k-mooney | mnaser: but that willl only happen after the heartbeat detects the connection is broken | |
| 01:04:23 | sean-k-mooney | or you try to send an rpc form the compute node | |
| 01:05:37 | mnaser | Failed to declare consumer for topic 'compute': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute' in vhost '/nova' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute' in vhost '/nova' due to timeout | |
| 01:05:37 | sean-k-mooney | mnaser: i think the issue is the api is not ensuring the queue exists before calling the compute node | |
| 01:06:02 | mnaser | should the api ensure it exists though, or should the service recreate it when recovering? | |
| 01:06:13 | sean-k-mooney | why not both | |
| 01:06:23 | sapd1 | sean-k-mooney, ah. I put some key/value in properties option of each instance (our monitoring service), I have tried write a function to generate these properties when instances are created. But I want to update these properties after that. | |
| 01:06:36 | sean-k-mooney | the api could ensure it exists so that the message is there for when the compute connects | |
| 01:07:15 | sean-k-mooney | sapd1: right ... | |
| 01:07:49 | sean-k-mooney | sapd1: virsh edit is your best bet but when you say you added a fucntion do you mean in the nova code | |
| 01:08:31 | sean-k-mooney | because as im sure you know we delete and recreate teh domain every time we start/stop/hard-reboot the instance | |
| 01:09:00 | sean-k-mooney | so if you did not modify nova those metadata values would be lost on basically any lifecycle operation | |
| 01:09:37 | sean-k-mooney | generally form monitoring you better just pulling the instance uuid form the metadata and doing the corralation based on that | |
| 01:10:19 | sapd1 | sean-k-mooney, I have handled these actions bro, I have modified this function to generate xml with properties (https://github.com/openstack/nova/blob/b0e71d80c3/nova/virt/libvirt/driver.py#L4271 | |
| 01:10:44 | sapd1 | then if the instance is rebooted or hard-rebooted, It's still ok. | |
| 01:11:18 | sapd1 | sean-k-mooney, in my case, this is a special requirement :D | |
| 01:11:48 | mnaser | operation queue.declare caused a channel exception not_found: failed to perform operation on queue 'scheduler.ctl2-xxxxxx.vexxhost.cn' in vhost '/nova' due to timeout | |
| 01:11:49 | sean-k-mooney | hehe ok ya that shoudl work. any reaon not to push the extra metadata upstream or is it specifc to your deployment | |
| 01:11:52 | mnaser | that's from the rabbitmq logs | |
| 01:12:09 | mnaser | it means that for some reason, its actually rabbitmq thats not lteting it declare? | |
| 01:12:43 | sean-k-mooney | mnaser: im not sure what it means by channel exception not_found but ill ask google | |
| 01:12:52 | mnaser | yeah im trying to ask google too | |
| 01:12:57 | sapd1 | sean-k-mooney, it's a specific for my deployment :) | |
| 01:13:58 | sean-k-mooney | mnaser: Binding a non-existing queue or a non-existing exchange will fail with a 404 NOT_FOUND error | |
| 01:14:23 | sean-k-mooney | https://www.rabbitmq.com/channels.html | |
| 01:14:47 | mnaser | i wonder if it tries to declare, fails, then binds, and fails, and then bombs out | |
| 01:18:02 | sapd1 | sean-k-mooney, Does my case related this function (https://github.com/openstack/nova/blob/6657339a7393226c72b9c8a293a6b92695ecb4cb/nova/virt/driver.py#L1374)? | |
| 01:18:27 | mnaser | bleh, i've been at this for 2.5h -- i dunno what to do at this point, its a bit too much | |
| 01:19:04 | mnaser | https://github.com/rabbitmq/rabbitmq-server/issues/959 | |
| 01:19:05 | sean-k-mooney | mnaser: i would proably set the quese to durable and enabel auto delete if this is an issue for you on production | |
| 01:19:07 | mnaser | i wonder if this has to do with this | |
| 01:19:20 | sean-k-mooney | at least untill we figure this out more direclty | |
| 01:21:50 | mnaser | yeah ill just work around it for now.. my brain is toast | |
| 01:22:11 | mnaser | and there you go, restarted them all, went into spawning state | |
| 01:22:26 | mnaser | i bet neutron is suffering from teh same thing | |
| 01:22:27 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1661510 | |
| 01:22:28 | openstack | Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 01:23:11 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.opendev.org/576689 | |
| 01:23:28 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.opendev.org/576709 | |
| 01:23:40 | sean-k-mooney | mnaser: i think https://bugs.launchpad.net/oslo.messaging/+bug/1661510 is basically the issue right | |
| 01:23:41 | openstack | Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 01:23:41 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.opendev.org/576712 | |
| 01:27:25 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1437955 seams to be also related | |
| 01:27:26 | openstack | Launchpad bug 1437955 in oslo.messaging "RPC calls and responses do not use the mandatory flag (AMQP)" [Wishlist,Confirmed] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 01:29:24 | mnaser | sean-k-mooney: that seems exactly it | |
| 01:30:47 | sean-k-mooney | mnaser: melwitt so i cuess we want to encourage https://review.opendev.org/#/c/660373/ | |