| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-04 | |||
| 17:25:00 | sean-k-mooney | the less agressive warning would be to warn on total_ram + swap | |
| 17:25:38 | sean-k-mooney | instead of > then total ram but once your are over total ram you are in OOM killer terrorty | |
| 17:26:58 | melwitt | sean-k-mooney: ah I see, thanks | |
| 17:27:23 | sean-k-mooney | melwitt: do you rememebr the nova bug for that transport thing i borught up at the end | |
| 17:27:42 | sean-k-mooney | melwitt: i remember debuging it with you at the time | |
| 17:27:52 | melwitt | yeah vaguely | |
| 17:28:01 | melwitt | it's the mandatory flag for oslo messaging | |
| 17:28:05 | sean-k-mooney | yes | |
| 17:28:28 | melwitt | I don't remember if we opened a bug about that specifically or what bug we referred to when talking about it | |
| 17:28:33 | sean-k-mooney | so it looks like there is an example of using it here | |
| 17:28:35 | sean-k-mooney | https://github.com/Gsantomaggio/rabbitmq-utils/blob/master/openstack/mandatory_test/mandatory_client_fail.py#L40-L54 | |
| 17:29:32 | sean-k-mooney | well i guess https://github.com/Gsantomaggio/rabbitmq-utils/blob/master/openstack/mandatory_test/mandatory_test.py#L35-L45 | |
| 17:29:36 | sean-k-mooney | is a better example | |
| 17:30:46 | sean-k-mooney | but what im not clear on is shoudl we just be setting it on every call or just when we do a topic send | |
| 17:31:16 | sean-k-mooney | the examples seam to be for topic sends | |
| 17:32:34 | melwitt | yeah, prob have the ask oslo peeps | |
| 17:32:40 | melwitt | cause I dunno either | |
| 17:33:35 | sean-k-mooney | i guess i could propose a patch and try adding it to things and see what breaks | |
| 17:33:42 | openstack | Launchpad bug 1849741 in oslo.messaging "TransportOptions are not documented" [Medium,Triaged] | |
| 17:33:42 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1849741 | |
| 17:33:44 | sean-k-mooney | that does not help however | |
| 17:34:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use 'Exception.__traceback__' for versioned notifications https://review.opendev.org/733668 | |
| 17:34:33 | openstack | bug 1881455 in OpenStack Compute (nova) "migrate server reporting list index out of bound" [Undecided,New] https://launchpad.net/bugs/1881455 | |
| 17:34:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reproducer for bug #1881455 https://review.opendev.org/733667 | |
| 17:39:41 | sean-k-mooney | stephenfin: by the way i don tthink you have to use Exception.__traceback__ directly anymore | |
| 17:39:51 | sean-k-mooney | infact that is an anti patteren i think | |
| 17:40:38 | openstack | Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,Incomplete] | |
| 17:40:38 | melwitt | sean-k-mooney: could it have been this? https://bugs.launchpad.net/nova/+bug/1854992 | |
| 17:40:56 | sean-k-mooney | melwitt: yep that was the nova bug | |
| 17:41:15 | melwitt | ok, can you write a note on it so we can have more hope to find it next time xD | |
| 17:41:15 | sean-k-mooney | well i think it was the second one | |
| 17:41:41 | melwitt | the only reason I found that was because I thought I remembered erik reporting it | |
| 17:42:02 | sean-k-mooney | yep erik is from blizzard right | |
| 17:42:06 | melwitt | yeah | |
| 17:42:15 | sean-k-mooney | options = oslo_messaging.TransportOptions(at_least_once=True) | |
| 17:42:17 | sean-k-mooney | client = oslo_messaging.RPCClient(transport, | |
| 17:42:20 | sean-k-mooney | target, | |
| 17:42:21 | sean-k-mooney | transport_options=options) | |
| 17:42:23 | sean-k-mooney | so i think that is all we have to do | |
| 17:43:08 | melwitt | I wonder if there's a way we could test that it does something | |
| 17:43:22 | sean-k-mooney | where we do this https://github.com/openstack/nova/blob/master/nova/rpc.py#L208-L212 | |
| 17:43:28 | sean-k-mooney | and well the other RPC clients | |
| 17:44:21 | sean-k-mooney | that is what https://github.com/Gsantomaggio/rabbitmq-utils/tree/master/openstack/mandatory_test is trying to do | |
| 17:45:08 | sean-k-mooney | when it fails to deliver a message to the queue | |
| 17:45:13 | sean-k-mooney | we will get a oslo_messaging.exceptions.MessageUndeliverable | |
| 17:45:18 | sean-k-mooney | exception | |
| 17:45:35 | melwitt | yeah I mean, how do you create that env where it would have raised MessageUndeliverable? | |
| 17:45:45 | melwitt | just stop rabbitmq process or? | |
| 17:46:50 | melwitt | just curious how to repro and see it fix the problem | |
| 17:46:58 | sean-k-mooney | no you need to send a message to a queue that does not exist | |
| 17:47:21 | melwitt | ok, I don't know how to do that I guess | |
| 17:47:51 | sean-k-mooney | we could create an exchange without any queue and jsut send a message to a random queue name | |
| 17:47:52 | melwitt | or is that the 'foo' in the example | |
| 17:48:06 | sean-k-mooney | ya so we can sed to foo | |
| 17:48:09 | sean-k-mooney | but not create foo | |
| 17:48:12 | melwitt | ok. shows how much I know about rabbitmq :P | |
| 17:48:29 | sean-k-mooney | well i think that is how this works | |
| 17:49:51 | melwitt | cool. I'm just thinking about a one-off local test just so we know we set the option in the right place and get the improved behavior. since the unit tests that go with the patch will only be checking "did you pass mandatory at this call site" | |
| 17:50:30 | sean-k-mooney | i wondering if we could do an opertunistic functional test or something | |
| 17:51:16 | sean-k-mooney | e.g. if rabbitmq is avaiable and we have a nova.conf with creads connect and try and to do cause the excption | |
| 17:51:32 | sean-k-mooney | i would assume/hope oslo messaging does that | |
| 17:52:23 | melwitt | yeah I dunno about that | |
| 17:52:38 | sean-k-mooney | i can hope | |
| 17:52:38 | melwitt | I didn't think we have rabbitmq available in those | |
| 17:52:46 | sean-k-mooney | we dont in nova | |
| 17:53:02 | sean-k-mooney | we use the inmemory oslo messaging driver | |
| 17:53:15 | sean-k-mooney | for our functional tests | |
| 17:54:02 | melwitt | yeah but that's the 'fake' driver. yeah I mean maybe it models that scenario? I was doubting | |
| 17:56:49 | sean-k-mooney | https://github.com/openstack/oslo.messaging/blob/e44c9883066d9b2d081a594b97aac3d598d491c9/oslo_messaging/tests/functional/test_functional.py#L154-L186 | |
| 17:57:46 | sean-k-mooney | so based on the fact that skips if its not rabbit:// | |
| 17:57:56 | sean-k-mooney | i would guess that it only works if its aviable | |
| 17:59:17 | melwitt | ok. besides that, that makes it look like we need to do more than just set the flag right? we have to do something to handle MessageUndeliverable | |
| 17:59:39 | sean-k-mooney | yes | |
| 17:59:46 | sean-k-mooney | which woudl depend on the rpc call | |
| 17:59:54 | sean-k-mooney | for spwan i guess put the vm to error state? | |
| 18:00:11 | sean-k-mooney | not really sure about what we woudl do for anything else | |
| 18:01:10 | melwitt | ah so this is more complicated than I thought | |
| 18:01:24 | melwitt | I thought all we'd have to do is set the flag and then oslo.messaging would reconnect us and retry or something like that | |
| 18:01:37 | sean-k-mooney | the issue is that the queue that the compute agent created nolonger exsits but the compute agent does not know that. so unless we had the compute agent call itself on that queue in a heart beat we would not know we had to recreate the queue | |
| 18:02:03 | sean-k-mooney | melwitt: im not sure if we can create the queue or if it has to be the compute agent | |
| 18:02:17 | sean-k-mooney | the compute agent normally creates the queue when it connect to rabbitmq | |
| 18:02:40 | sean-k-mooney | which is why restarting the compuate agent fixes the problem | |
| 18:03:23 | sean-k-mooney | so at a minium we can log that the agent is unreachable. we could even mark the agent as down but that leaves the question of who will make it up again once its restarted | |
| 18:03:31 | melwitt | yeah. well, then I'd wonder if we could put try-except in the nova/rpc layer that will create the queue if we get MessageUndeliverable? that way whoever gets the "no queue" state will recreate it? | |
| 18:04:27 | sean-k-mooney | if we had a periodic task in teh compute agent that sent to its own queue then that would work | |
| 18:04:38 | sean-k-mooney | it might work if the conductor or api created it too | |
| 18:05:00 | sean-k-mooney | i just dont know if the compute agent will start litening to the queue when its created by someone else | |
| 18:05:41 | melwitt | yeah I guess I'm thinking it wouldn't matter who creates the queue. like if nova-conductor gets MessageUndeliverable trying to talk to compute, assume the queue is gone and recreate it, and resend | |
| 18:05:47 | melwitt | oh, right | |
| 18:05:50 | melwitt | hm | |
| 18:06:22 | sean-k-mooney | that is why i was suggesting havign the compute agent send to its own queue | |
| 18:06:30 | sean-k-mooney | jsut a simple ping/heatbeat | |
| 18:06:40 | melwitt | periodic does sound like it would work but that doesn't help someone trying to boot an instance and it fails due to this | |
| 18:07:01 | melwitt | but of course better than what we have today | |
| 18:07:02 | sean-k-mooney | we could retry after a short interval? | |
| 18:07:13 | sean-k-mooney | hoping the agent fixes its self? | |
| 18:08:08 | sean-k-mooney | maybe if we have time tommorow and ben or some of the oslo folks are aroudn we could ask them | |
| 18:08:36 | sean-k-mooney | our i guess i could try it locally | |
| 18:08:44 | melwitt | yeah... I don't love that idea but yeah, would take some thought on some options | |
| 18:08:51 | sean-k-mooney | i could use the rabbitmq gui to delete the queue manually | |