| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-06 | |||
| 23:35:52 | melwitt | if the super conductor receives the reply, the next step is super conductor casts to the cell mq | |
| 23:36:05 | mnaser | ok, ill move away from scheduler and start looking at conductor now | |
| 23:37:12 | mnaser | conductor, the worlds quietest service | |
| 23:37:16 | melwitt | :) | |
| 23:38:12 | melwitt | you know.... we do cache mq connection info. so if you find it's hanging up at the step where super conductor is trying to drop the "build instance" message onto the cell mq and that cached info is no longer valid, that could be the culprit | |
| 23:38:40 | melwitt | but no, you said things return to normal if you kill the rabbit queues and recreate them without restarting any nova services yeah? | |
| 23:38:54 | melwitt | that would mean the nova cache thing doesn't matter | |
| 23:40:23 | mnaser | ok so schedule_and_build_instances gets to the point where its gets an asigned host | |
| 23:42:52 | mnaser | and it creates a block device mapping in the db for the cell | |
| 23:43:56 | melwitt | ok, so it's getting the reply from the scheduler. so it must be that the message going onto the cell mq is going into the ether | |
| 23:43:59 | melwitt | which makes me suspicious about the mq connection caching | |
| 23:44:43 | melwitt | mnaser: when you do the queues recreate thing, are you restarting nova services after? | |
| 23:45:16 | mnaser | yes i have to restart them too after (but i do that just to keep things clean rather than make it work) | |
| 23:45:21 | melwitt | *do you restart nova services after? | |
| 23:45:33 | melwitt | ok, then it might be a problem with that cache (it never refreshes) | |
| 23:46:05 | melwitt | have you ever tried restarting services without recreating the queues? | |
| 23:46:40 | mnaser | yep, no bueno | |
| 23:47:18 | melwitt | restarting would only help new server creates though, not the stuck ones | |
| 23:47:19 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.opendev.org/637716 | |
| 23:47:51 | mnaser | so i know i at least get to here: https://github.com/openstack/nova/blob/3370f0f03ce17aaf3a7ebaa95d497f62bef238c0/nova/conductor/manager.py#L1400-L1401 | |
| 23:48:01 | melwitt | mnaser: does recreating the queues and restarting services make the stuck servers finish building? | |
| 23:48:05 | mnaser | melwitt: yeah but ive restarted them all in this case and no bueno | |
| 23:48:06 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.opendev.org/641200 | |
| 23:48:11 | mnaser | nope, all the stuck ones stay stuck | |
| 23:48:26 | melwitt | ok, so restarting services only will NOT make new server creates work? | |
| 23:48:35 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020 | |
| 23:48:58 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027 | |
| 23:49:07 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031 | |
| 23:49:52 | mnaser | melwitt: correct | |
| 23:49:54 | melwitt | if restarting services only doesn't make new server creates work, then I'd think the issue has to be with rabbit or oslo.messaging | |
| 23:50:08 | mnaser | yeah my feeling is this is oslo.messaging | |
| 23:50:35 | mnaser | https://github.com/openstack/nova/blob/3370f0f03ce17aaf3a7ebaa95d497f62bef238c0/nova/conductor/manager.py#L1462-L1473 -- i get here, i see it connect to rabbit, i see it cast on compute. |
|
| 23:50:37 | mnaser | and then nothing | |
| 23:51:05 | melwitt | could also be a mirrored queues thing in rabbit, there's been that similar bug in the past | |
| 23:51:13 | melwitt | that I linked earlier | |
| 23:51:39 | mnaser | yeah.. it also connects to rabbitmq successfuly... well the weird thing is i think maybe what is broken is casts only? | |
| 23:51:46 | mnaser | because the normal rpc stuff works | |
| 23:51:51 | mnaser | its the compute cast that fails | |
| 23:52:22 | melwitt | you could try something that is a call to compute to see if that works. that would be an interesting test | |
| 23:52:28 | melwitt | like getting vnc console url | |
| 23:52:28 | mnaser | the service does report as up too inside nova | |
| 23:52:33 | mnaser | dont have any vms unfortunately :P | |
| 23:52:44 | melwitt | lemme see if there's something else | |
| 23:52:56 | mnaser | hypervisor-stats maybe | |
| 23:53:11 | melwitt | get_diagnostics | |
| 23:53:17 | mnaser | err -utpime | |
| 23:53:51 | mnaser | isnt diag per vm | |
| 23:54:19 | mnaser | nova hypervisor-uptime <hypervisor-uuid> hangs | |
| 23:54:29 | melwitt | oh yeah, sorry | |
| 23:54:41 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Specifying az when restore shelved server https://review.opendev.org/624689 | |
| 23:54:51 | melwitt | ok, so it doesn't have to do with cast vs call, it's an issue with trying to get to the cell message queue in general | |
| 23:56:26 | mnaser | i mean the configs match.. in terms of cell_mapping | |
| 23:56:31 | mnaser | and it successfully connects.. | |
| 23:56:46 | mnaser | i guess now that i know its just reproducable with something far more basic.. | |
| 23:56:56 | melwitt | yeah... true | |
| 23:58:29 | sean-k-mooney | melwitt: are you suspecting this is is similar to the wsgi issue? | |
| 23:59:02 | mnaser | CALL msg_id: 7dc9fa9615ed4ef8bf083ac84e40beef exchange 'nova' topic 'compute.<hostname>' _send | |
| 23:59:19 | melwitt | sean-k-mooney: no, this is something different. I wondered if it's something like this old bug that was fixed in the past, if another race like this has re-emerged https://github.com/rabbitmq/rabbitmq-server/issues/224 | |
| 23:59:26 | mnaser | that's how far it'll get and then hang, same way it hung for the new compute | |
| 23:59:42 | mnaser | i mean let me restart the nova-compute... but still feels like it should be recovering on its own.. | |
| #openstack-nova - 2019-06-07 | |||
| 00:00:06 | mnaser | i mean | |
| 00:00:13 | melwitt | sean-k-mooney: well, at least I didn't think it's related, but wasn't thinking in that direction at all tbh | |
| 00:00:28 | mnaser | oh this is interesting | |
| 00:00:33 | mnaser | the compute service is getting a messagingtimeout | |
| 00:00:36 | mnaser | even though conductor is up | |
| 00:00:43 | melwitt | yeah, it should be recovering on its own | |
| 00:00:58 | sean-k-mooney | melwitt: ok i just wondered since you were suggesting the vnc console call | |
| 00:01:02 | mnaser | just looping on "Timed out waiting for reply.." | |
| 00:01:17 | melwitt | sean-k-mooney: haha no, we were just trying to see if the cast vs call behavior was consistent | |
| 00:01:22 | sean-k-mooney | just got back to the start of the converstation so now have a bit more context | |
| 00:01:37 | mnaser | i'll try restarting it (i have bunch of other computes that are broken so at least it will say if a restart fixes it) | |
| 00:01:47 | mnaser | ill still have a one in broken state | |
| 00:02:25 | melwitt | good thing you have a lot of broken computes huh! | |
| 00:02:29 | mnaser | hah | |
| 00:02:39 | mnaser | looks like it's not going up, the capabilities xml string just got logged and its stuck | |
| 00:02:42 | sean-k-mooney | we had an upstream or posible downstream bug that was similar to this | |
| 00:02:49 | sean-k-mooney | not to long ago | |
| 00:03:02 | mnaser | i assume it'll just blow up now complaining the message timed out | |
| 00:03:13 | melwitt | hm, weird | |
| 00:03:38 | melwitt | sean-k-mooney: lettuce know if you have any links | |
| 00:04:53 | mnaser | sigh. | |
| 00:04:56 | mnaser | two restarts and it works now. | |
| 00:05:03 | mnaser | or 3 rather | |
| 00:05:10 | mnaser | (that specific compute node works rather) | |
| 00:05:37 | mnaser | the rest still broken | |
| 00:05:46 | melwitt | 3 restarts made it work but not 1? | |
| 00:05:55 | melwitt | ... | |
| 00:06:06 | mnaser | yep, first restart it was hung | |
| 00:06:19 | melwitt | O.o | |
| 00:06:19 | mnaser | restarted once with debug enabled and accidnetally log levels without an s, restarted again right away with an s and it went up fine. | |
| 00:06:26 | mnaser | wait | |
| 00:06:42 | mnaser | it might actually have been okay | |
| 00:06:44 | mnaser | the first time :\ | |
| 00:07:14 | melwitt | ok, that is the only thing that could make sense there otherwise....mind blown | |
| 00:07:16 | mnaser | let me see whta happened with th broken ones | |
| 00:07:37 | sean-k-mooney | hehe if its a case that debug mode fixes it calling dibs on not finding out why :P | |
| 00:08:02 | mnaser | hmm, looks like its been doing the same thing | |
| 00:08:06 | mnaser | looping forever in messagingtimeout | |
| 00:08:08 | mnaser | without recover | |
| 00:08:57 | mnaser | it looks like its flapping | |