| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-04 | |||
| 17:32:34 | melwitt | yeah, prob have the ask oslo peeps | |
| 17:32:40 | melwitt | cause I dunno either | |
| 17:33:35 | sean-k-mooney | i guess i could propose a patch and try adding it to things and see what breaks | |
| 17:33:42 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1849741 | |
| 17:33:42 | openstack | Launchpad bug 1849741 in oslo.messaging "TransportOptions are not documented" [Medium,Triaged] | |
| 17:33:44 | sean-k-mooney | that does not help however | |
| 17:34:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reproducer for bug #1881455 https://review.opendev.org/733667 | |
| 17:34:33 | openstack | bug 1881455 in OpenStack Compute (nova) "migrate server reporting list index out of bound" [Undecided,New] https://launchpad.net/bugs/1881455 | |
| 17:34:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use 'Exception.__traceback__' for versioned notifications https://review.opendev.org/733668 | |
| 17:39:41 | sean-k-mooney | stephenfin: by the way i don tthink you have to use Exception.__traceback__ directly anymore | |
| 17:39:51 | sean-k-mooney | infact that is an anti patteren i think | |
| 17:40:38 | melwitt | sean-k-mooney: could it have been this? https://bugs.launchpad.net/nova/+bug/1854992 | |
| 17:40:38 | openstack | Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,Incomplete] | |
| 17:40:56 | sean-k-mooney | melwitt: yep that was the nova bug | |
| 17:41:15 | sean-k-mooney | well i think it was the second one | |
| 17:41:15 | melwitt | ok, can you write a note on it so we can have more hope to find it next time xD | |
| 17:41:41 | melwitt | the only reason I found that was because I thought I remembered erik reporting it | |
| 17:42:02 | sean-k-mooney | yep erik is from blizzard right | |
| 17:42:06 | melwitt | yeah | |
| 17:42:15 | sean-k-mooney | options = oslo_messaging.TransportOptions(at_least_once=True) | |
| 17:42:17 | sean-k-mooney | client = oslo_messaging.RPCClient(transport, | |
| 17:42:20 | sean-k-mooney | target, | |
| 17:42:21 | sean-k-mooney | transport_options=options) | |
| 17:42:23 | sean-k-mooney | so i think that is all we have to do | |
| 17:43:08 | melwitt | I wonder if there's a way we could test that it does something | |
| 17:43:22 | sean-k-mooney | where we do this https://github.com/openstack/nova/blob/master/nova/rpc.py#L208-L212 | |
| 17:43:28 | sean-k-mooney | and well the other RPC clients | |
| 17:44:21 | sean-k-mooney | that is what https://github.com/Gsantomaggio/rabbitmq-utils/tree/master/openstack/mandatory_test is trying to do | |
| 17:45:08 | sean-k-mooney | when it fails to deliver a message to the queue | |
| 17:45:13 | sean-k-mooney | we will get a oslo_messaging.exceptions.MessageUndeliverable | |
| 17:45:18 | sean-k-mooney | exception | |
| 17:45:35 | melwitt | yeah I mean, how do you create that env where it would have raised MessageUndeliverable? | |
| 17:45:45 | melwitt | just stop rabbitmq process or? | |
| 17:46:50 | melwitt | just curious how to repro and see it fix the problem | |
| 17:46:58 | sean-k-mooney | no you need to send a message to a queue that does not exist | |
| 17:47:21 | melwitt | ok, I don't know how to do that I guess | |
| 17:47:51 | sean-k-mooney | we could create an exchange without any queue and jsut send a message to a random queue name | |
| 17:47:52 | melwitt | or is that the 'foo' in the example | |
| 17:48:06 | sean-k-mooney | ya so we can sed to foo | |
| 17:48:09 | sean-k-mooney | but not create foo | |
| 17:48:12 | melwitt | ok. shows how much I know about rabbitmq :P | |
| 17:48:29 | sean-k-mooney | well i think that is how this works | |
| 17:49:51 | melwitt | cool. I'm just thinking about a one-off local test just so we know we set the option in the right place and get the improved behavior. since the unit tests that go with the patch will only be checking "did you pass mandatory at this call site" | |
| 17:50:30 | sean-k-mooney | i wondering if we could do an opertunistic functional test or something | |
| 17:51:16 | sean-k-mooney | e.g. if rabbitmq is avaiable and we have a nova.conf with creads connect and try and to do cause the excption | |
| 17:51:32 | sean-k-mooney | i would assume/hope oslo messaging does that | |
| 17:52:23 | melwitt | yeah I dunno about that | |
| 17:52:38 | melwitt | I didn't think we have rabbitmq available in those | |
| 17:52:38 | sean-k-mooney | i can hope | |
| 17:52:46 | sean-k-mooney | we dont in nova | |
| 17:53:02 | sean-k-mooney | we use the inmemory oslo messaging driver | |
| 17:53:15 | sean-k-mooney | for our functional tests | |
| 17:54:02 | melwitt | yeah but that's the 'fake' driver. yeah I mean maybe it models that scenario? I was doubting | |
| 17:56:49 | sean-k-mooney | https://github.com/openstack/oslo.messaging/blob/e44c9883066d9b2d081a594b97aac3d598d491c9/oslo_messaging/tests/functional/test_functional.py#L154-L186 | |
| 17:57:46 | sean-k-mooney | so based on the fact that skips if its not rabbit:// | |
| 17:57:56 | sean-k-mooney | i would guess that it only works if its aviable | |
| 17:59:17 | melwitt | ok. besides that, that makes it look like we need to do more than just set the flag right? we have to do something to handle MessageUndeliverable | |
| 17:59:39 | sean-k-mooney | yes | |
| 17:59:46 | sean-k-mooney | which woudl depend on the rpc call | |
| 17:59:54 | sean-k-mooney | for spwan i guess put the vm to error state? | |
| 18:00:11 | sean-k-mooney | not really sure about what we woudl do for anything else | |
| 18:01:10 | melwitt | ah so this is more complicated than I thought | |
| 18:01:24 | melwitt | I thought all we'd have to do is set the flag and then oslo.messaging would reconnect us and retry or something like that | |
| 18:01:37 | sean-k-mooney | the issue is that the queue that the compute agent created nolonger exsits but the compute agent does not know that. so unless we had the compute agent call itself on that queue in a heart beat we would not know we had to recreate the queue | |
| 18:02:03 | sean-k-mooney | melwitt: im not sure if we can create the queue or if it has to be the compute agent | |
| 18:02:17 | sean-k-mooney | the compute agent normally creates the queue when it connect to rabbitmq | |
| 18:02:40 | sean-k-mooney | which is why restarting the compuate agent fixes the problem | |
| 18:03:23 | sean-k-mooney | so at a minium we can log that the agent is unreachable. we could even mark the agent as down but that leaves the question of who will make it up again once its restarted | |
| 18:03:31 | melwitt | yeah. well, then I'd wonder if we could put try-except in the nova/rpc layer that will create the queue if we get MessageUndeliverable? that way whoever gets the "no queue" state will recreate it? | |
| 18:04:27 | sean-k-mooney | if we had a periodic task in teh compute agent that sent to its own queue then that would work | |
| 18:04:38 | sean-k-mooney | it might work if the conductor or api created it too | |
| 18:05:00 | sean-k-mooney | i just dont know if the compute agent will start litening to the queue when its created by someone else | |
| 18:05:41 | melwitt | yeah I guess I'm thinking it wouldn't matter who creates the queue. like if nova-conductor gets MessageUndeliverable trying to talk to compute, assume the queue is gone and recreate it, and resend | |
| 18:05:47 | melwitt | oh, right | |
| 18:05:50 | melwitt | hm | |
| 18:06:22 | sean-k-mooney | that is why i was suggesting havign the compute agent send to its own queue | |
| 18:06:30 | sean-k-mooney | jsut a simple ping/heatbeat | |
| 18:06:40 | melwitt | periodic does sound like it would work but that doesn't help someone trying to boot an instance and it fails due to this | |
| 18:07:01 | melwitt | but of course better than what we have today | |
| 18:07:02 | sean-k-mooney | we could retry after a short interval? | |
| 18:07:13 | sean-k-mooney | hoping the agent fixes its self? | |
| 18:08:08 | sean-k-mooney | maybe if we have time tommorow and ben or some of the oslo folks are aroudn we could ask them | |
| 18:08:36 | sean-k-mooney | our i guess i could try it locally | |
| 18:08:44 | melwitt | yeah... I don't love that idea but yeah, would take some thought on some options | |
| 18:08:51 | sean-k-mooney | i could use the rabbitmq gui to delete the queue manually | |
| 18:09:49 | sean-k-mooney | i dont think the manamge ui is isntall by devstack by default but i dont think its that hard to enable | |
| 18:10:15 | sean-k-mooney | if i could repoduce it manually then we could test teh recreate behavior | |
| 18:10:36 | sean-k-mooney | e.g. if i make the condutor recreate it and i still can boot a vm i know it need to be the agent | |
| 18:10:41 | melwitt | right | |
| 18:11:53 | melwitt | yeah doing that would demystify a lot. but yeah gonna be a pain/not ideal if nova-compute would need to re-init to pick up the new queue | |
| 18:14:11 | sean-k-mooney | im going to go figure out what im doing for dinner and then i might give it a try although i kind of want to look into it someother time | |
| 18:14:28 | sean-k-mooney | can you bug me about this if you remember tommorow or next week | |
| 18:14:54 | bnemec | If you have messaging stuff to talk about I would suggest pinging kgiusti to make sure he's available. | |
| 18:14:56 | melwitt | sure. I'll try to write something on the launchpad bug so I don't forget about this a sixth time or whatever we're at now | |
| 18:15:22 | melwitt | in addition | |
| 18:15:45 | sean-k-mooney | bnemec: well we are just trying to figure out how to use the mandataory flag that ye enabled via the transport options correctly | |
| 18:16:16 | sean-k-mooney | but my rabbit mq knoladge is really not good enough to have an intuition about this | |
| 18:16:20 | melwitt | use it + handle the exception that will raise as a result of setting it | |
| 18:16:38 | melwitt | *set it | |
| 18:17:14 | sean-k-mooney | melwitt: hehe yep 6th might be a bit much but at least thrid or forth :) | |