| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-07 | |||
| 00:51:13 | sean-k-mooney | actully i think its via this https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/service.py#L184-L192 | |
| 00:51:25 | mnaser | https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/amqpdriver.py#L650-L665 | |
| 00:52:25 | sean-k-mooney | yes so https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/service.py#L186 | |
| 00:52:52 | sean-k-mooney | is where in the base nova serivce calse we declar the target for an rpc to the service based on the topic and host | |
| 00:54:04 | mnaser | i think that is not being recreated.. | |
| 00:54:59 | openstackgerrit | Daniel Speichert proposed openstack/nova-specs master: Directly download and upload images to RBD https://review.opendev.org/658903 | |
| 00:55:09 | sean-k-mooney | i think so too | |
| 00:55:28 | sean-k-mooney | it intally gets created here https://github.com/openstack/nova/blob/013aa1915c79cfcb90c4333ce1e16b3c40f16be8/nova/cmd/compute.py#L56-L58 | |
| 00:55:52 | sean-k-mooney | when we start the agent | |
| 00:56:00 | mnaser | and i think that ends up all the way down here when it starts listneing https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/amqpdriver.py#L650-L665 | |
| 00:56:15 | mnaser | but declare_topic_consumer doesnt store state or anything, it just does it one time | |
| 00:56:29 | mnaser | so if the queue disappears, declare_topic_consumer technically never runs | |
| 00:57:35 | sean-k-mooney | right so we need to add it to our consumers | |
| 00:58:37 | sean-k-mooney | that said | |
| 00:58:39 | sean-k-mooney | https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L946-L969 | |
| 00:58:55 | sean-k-mooney | i think that shoudl be happening already | |
| 01:00:20 | sean-k-mooney | so set_current_channel shoudl redeclar it here https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L792-L793 | |
| 01:01:22 | sapd1 | Hi everyone, I would like to regenerate metadata/properties in libvirt xml (I mean virsh dumpxml output) with running instances. How can I do that without reboot instances :D | |
| 01:01:29 | sean-k-mooney | we do not call self._set_current_channel here however https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/impl_rabbit.py#L756 | |
| 01:02:01 | sean-k-mooney | sapd1: you cant really what are you actully trying to do | |
| 01:02:02 | mnaser | i didn't hit that ecxeption i think | |
| 01:03:16 | sean-k-mooney | sapd1: i mean you can update the xml with virsh edit but im not sure why you want to modify the metadata in the xml | |
| 01:03:48 | sean-k-mooney | mnaser: ya if you are hitting the generic exception below that you shoudl end up redeclarign the queue | |
| 01:04:12 | sean-k-mooney | mnaser: but that willl only happen after the heartbeat detects the connection is broken | |
| 01:04:23 | sean-k-mooney | or you try to send an rpc form the compute node | |
| 01:05:37 | mnaser | Failed to declare consumer for topic 'compute': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute' in vhost '/nova' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'compute' in vhost '/nova' due to timeout | |
| 01:05:37 | sean-k-mooney | mnaser: i think the issue is the api is not ensuring the queue exists before calling the compute node | |
| 01:06:02 | mnaser | should the api ensure it exists though, or should the service recreate it when recovering? | |
| 01:06:13 | sean-k-mooney | why not both | |
| 01:06:23 | sapd1 | sean-k-mooney, ah. I put some key/value in properties option of each instance (our monitoring service), I have tried write a function to generate these properties when instances are created. But I want to update these properties after that. | |
| 01:06:36 | sean-k-mooney | the api could ensure it exists so that the message is there for when the compute connects | |
| 01:07:15 | sean-k-mooney | sapd1: right ... | |
| 01:07:49 | sean-k-mooney | sapd1: virsh edit is your best bet but when you say you added a fucntion do you mean in the nova code | |
| 01:08:31 | sean-k-mooney | because as im sure you know we delete and recreate teh domain every time we start/stop/hard-reboot the instance | |
| 01:09:00 | sean-k-mooney | so if you did not modify nova those metadata values would be lost on basically any lifecycle operation | |
| 01:09:37 | sean-k-mooney | generally form monitoring you better just pulling the instance uuid form the metadata and doing the corralation based on that | |
| 01:10:19 | sapd1 | sean-k-mooney, I have handled these actions bro, I have modified this function to generate xml with properties (https://github.com/openstack/nova/blob/b0e71d80c3/nova/virt/libvirt/driver.py#L4271 | |
| 01:10:44 | sapd1 | then if the instance is rebooted or hard-rebooted, It's still ok. | |
| 01:11:18 | sapd1 | sean-k-mooney, in my case, this is a special requirement :D | |
| 01:11:48 | mnaser | operation queue.declare caused a channel exception not_found: failed to perform operation on queue 'scheduler.ctl2-xxxxxx.vexxhost.cn' in vhost '/nova' due to timeout | |
| 01:11:49 | sean-k-mooney | hehe ok ya that shoudl work. any reaon not to push the extra metadata upstream or is it specifc to your deployment | |
| 01:11:52 | mnaser | that's from the rabbitmq logs | |
| 01:12:09 | mnaser | it means that for some reason, its actually rabbitmq thats not lteting it declare? | |
| 01:12:43 | sean-k-mooney | mnaser: im not sure what it means by channel exception not_found but ill ask google | |
| 01:12:52 | mnaser | yeah im trying to ask google too | |
| 01:12:57 | sapd1 | sean-k-mooney, it's a specific for my deployment :) | |
| 01:13:58 | sean-k-mooney | mnaser: Binding a non-existing queue or a non-existing exchange will fail with a 404 NOT_FOUND error | |
| 01:14:23 | sean-k-mooney | https://www.rabbitmq.com/channels.html | |
| 01:14:47 | mnaser | i wonder if it tries to declare, fails, then binds, and fails, and then bombs out | |
| 01:18:02 | sapd1 | sean-k-mooney, Does my case related this function (https://github.com/openstack/nova/blob/6657339a7393226c72b9c8a293a6b92695ecb4cb/nova/virt/driver.py#L1374)? | |
| 01:18:27 | mnaser | bleh, i've been at this for 2.5h -- i dunno what to do at this point, its a bit too much | |
| 01:19:04 | mnaser | https://github.com/rabbitmq/rabbitmq-server/issues/959 | |
| 01:19:05 | sean-k-mooney | mnaser: i would proably set the quese to durable and enabel auto delete if this is an issue for you on production | |
| 01:19:07 | mnaser | i wonder if this has to do with this | |
| 01:19:20 | sean-k-mooney | at least untill we figure this out more direclty | |
| 01:21:50 | mnaser | yeah ill just work around it for now.. my brain is toast | |
| 01:22:11 | mnaser | and there you go, restarted them all, went into spawning state | |
| 01:22:26 | mnaser | i bet neutron is suffering from teh same thing | |
| 01:22:27 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1661510 | |
| 01:22:28 | openstack | Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 01:23:11 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.opendev.org/576689 | |
| 01:23:28 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.opendev.org/576709 | |
| 01:23:40 | sean-k-mooney | mnaser: i think https://bugs.launchpad.net/oslo.messaging/+bug/1661510 is basically the issue right | |
| 01:23:41 | openstack | Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 01:23:41 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.opendev.org/576712 | |
| 01:27:25 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1437955 seams to be also related | |
| 01:27:26 | openstack | Launchpad bug 1437955 in oslo.messaging "RPC calls and responses do not use the mandatory flag (AMQP)" [Wishlist,Confirmed] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 01:29:24 | mnaser | sean-k-mooney: that seems exactly it | |
| 01:30:47 | sean-k-mooney | mnaser: melwitt so i cuess we want to encourage https://review.opendev.org/#/c/660373/ | |
| 01:45:10 | mnaser | sean-k-mooney: I guess so! | |
| 01:45:50 | sean-k-mooney | the current approch would require nova to be modified to handel the retry whic may or may not be what we want | |
| 01:46:06 | sean-k-mooney | i left some feed back in the review | |
| 01:46:46 | sean-k-mooney | but its shoudl help even if we need to do some extra work as a consumer of oslo.messaging | |
| 01:47:52 | sean-k-mooney | mnaser: i was tired and when to sleep for 4 hours so while its 2 am from me here it must be getting pretty late for you | |
| 01:48:18 | sean-k-mooney | well almost 3 but as i said i alreay got some sleep | |
| 01:54:43 | mnaser | sean-k-mooney: where are you based out of? | |
| 01:54:52 | sean-k-mooney | ireland | |
| 01:55:10 | sean-k-mooney | but i tend to shift my day to overlap with the us more | |
| 01:57:52 | sean-k-mooney | i would normally end my day and go to sleep at aound 1 am but i was tired and went to bed at 8pm today but when i do that i never get more then about 4 hours and then i get up and work for a bit and typically go back to sleep around 3-4 ish | |
| 01:58:11 | sean-k-mooney | or 8-9 ish if my sleeping pattern is totally messed up | |
| 03:28:23 | openstackgerrit | Daniel Speichert proposed openstack/nova-specs master: Directly download and upload images to RBD https://review.opendev.org/658903 | |
| 04:32:17 | openstackgerrit | Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.opendev.org/643664 | |
| 04:32:18 | openstackgerrit | Eric Fried proposed openstack/nova master: Use OpenStack SDK for placement https://review.opendev.org/656023 | |
| 07:20:03 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP:Specify availability_zone to unshelve https://review.opendev.org/663851 | |
| 07:55:27 | kashyap | sean-k-mooney: Yeah, I _did_ see the obscure "vm_mode" yesterday, and reading its documentation. It is a Xen-related term, but used as a general abstraction in Nova to represent any hypervisor | |
| 07:56:11 | kashyap | tonyb: I wanted to do some tests on AArch64 host; got it from another dev. | |
| 07:56:44 | kashyap | efried: No, that "pick guest arch" Blueprint is not for Train. (It's Tech Debt) | |
| 08:39:51 | tonyb[m] | kashyap: okay have fun | |
| 09:50:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fold ComputeManager._resize to its only caller https://review.opendev.org/663873 | |
| 10:23:37 | stephenfin | RuntimeError: On Python 3, zvm sdk supports to Python 3.6 | |
| 10:23:45 | stephenfin | and all I want to do is build docs | |
| 10:24:14 | stephenfin | We need to find some way to move that out of requirements.txt. More hassle than it's worth. | |
| 10:37:33 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: docs: Don't version links to reno docs https://review.opendev.org/662178 | |
| 10:37:34 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: tox: Use basepython of python3.5 https://review.opendev.org/663887 | |
| 10:37:34 | openstackgerrit | Stephen Finucane proposed openstack/nova stable/rocky: tox: Stop build *all* docs in 'docs' https://review.opendev.org/663888 | |
| 11:40:18 | sean-k-mooney | just an fyi but i have marked https://blueprints.launchpad.net/nova/+spec/libvirt-neutron-sriov-livemigration as implemented/complete since the documentation update we were waiting for is now merged in neutron. | |
| 11:48:01 | openstackgerrit | Theodoros Tsioutsias proposed openstack/nova-specs master: Enable rebuild for instances in cell0 https://review.opendev.org/648686 | |
| 12:28:57 | openstackgerrit | Merged openstack/nova master: Make nova-next archive using --before https://review.opendev.org/661002 | |
| 12:35:47 | openstackgerrit | Merged openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.opendev.org/641806 | |
| 12:35:56 | openstackgerrit | Merged openstack/nova master: Robustify attachment tracking in CinderFixtureNewAttachFlow https://review.opendev.org/658904 | |