| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-17 | |||
| 16:13:43 | sean-k-mooney | well my point was the bottel neck has never been writing the code. it has alwasy been reviewing it. we can reduce the scope for new feature but i know that that leads to less in vestment in openstck | |
| 16:13:55 | bauzas | if I were bragging, that would be because I feel we got a consensus on the design we gonna achieve for the thing I wanna implement | |
| 16:14:10 | bauzas | but certainly not implying that my stuff is done | |
| 16:15:07 | bauzas | anyway, I need to call it a day | |
| 16:15:25 | bauzas | things change, people become parents of kids who grown up and have social activities | |
| 16:15:30 | bauzas | grow* | |
| 16:15:52 | bauzas | and since stupidely kids under 8 can't drive, I need to AWOL | |
| 16:16:47 | sean-k-mooney | bauzas: dont sell your kids short. have you given the opertuity to try. what could possibly go wrong | |
| 16:16:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete source allocations in move_allocations if target no longer exists https://review.opendev.org/689049 | |
| 16:20:16 | mriedem | a wild idea: more cores should do more reviews | |
| 16:20:19 | mriedem | *gasp* | |
| 16:20:48 | mriedem | https://www.stackalytics.com/report/contribution/nova/120 | |
| 16:20:51 | mriedem | who is <1 review per day on that list? | |
| 16:22:47 | efried | 1/3 of the core team. | |
| 16:22:59 | mriedem | right, | |
| 16:23:11 | mriedem | so if you're a core below that threshold, stop complaining | |
| 16:23:29 | efried | to be fair, I don't see those cores complaining. | |
| 16:23:39 | efried | Actually, I think I'm the only one complaining. | |
| 16:23:44 | mriedem | i see bauzas and melwitt complaining above | |
| 16:24:00 | melwitt | my comment was not intended as a complaint | |
| 16:25:04 | efried | fwiw I saw both as stating reasons for preferring the status quo wrt approving more than we can hope to review. | |
| 16:25:46 | dansmith | yeah, that was my understanding as well | |
| 16:26:12 | dansmith | both melwitt and bauzas have been around since we've tried many similar schemes in the past too | |
| 16:26:13 | dansmith | as have I and mriedem | |
| 16:26:47 | melwitt | I've not argued or voted on any of the new process things because I am in a difficult spot these days with upstream review time. I said one sentence from the perspective of being a contributor. I didn't want anyone to see it as complaining from me | |
| 16:32:45 | bauzas | folks, I was on and off last cycles, and I promised too much so now I'm done with this | |
| 16:32:59 | bauzas | what I just want is helping others as much as I can | |
| 16:33:21 | efried | There was a request for a feature liaison earlier :) | |
| 16:33:37 | bauzas | so, I'm glad mriedem pings me with asking to review stable changes for example, or spec review request | |
| 16:33:40 | efried | http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-10-17.log.html#t2019-10-17T04:14:06 | |
| 16:33:58 | bauzas | and then, if I can commit myself, I do | |
| 16:39:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Make '_wait_for_state_change' behave consistently https://review.opendev.org/689180 | |
| 16:39:12 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Unify '_wait_until_deleted' implementations https://review.opendev.org/689181 | |
| 16:39:12 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Make 'ServerTestBase' subclass 'InstanceHelperMixin' https://review.opendev.org/689182 | |
| 16:45:54 | eandersson | sean-k-mooney, the issue is oslo messaging related btw, or maybe rabbitmq related. | |
| 16:46:31 | eandersson | I tried to publish and consume to the specific compute queues and every thing worked fine | |
| 16:46:40 | eandersson | I even captured the message last night from the scheduler | |
| 16:46:52 | eandersson | but deleting the queues and restarting the compute, and magically it's working | |
| 17:00:45 | sean-k-mooney | that makes me think of an oslo bug | |
| 17:01:25 | sean-k-mooney | eandersson: https://bugs.launchpad.net/oslo.messaging/+bug/1661510 | |
| 17:01:25 | openstack | Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 17:01:37 | sean-k-mooney | i think that the one im thinking of | |
| 17:05:50 | sean-k-mooney | melwitt: do you rembere this oslo chage that intoduce the mandatory flag for rabbitmq | |
| 17:05:52 | sean-k-mooney | https://review.opendev.org/#/c/660373/ | |
| 17:06:05 | melwitt | a little bit, yeah | |
| 17:06:09 | sean-k-mooney | melwitt: do you recal if we ever started using it in nova | |
| 17:06:18 | melwitt | not that I know of | |
| 17:06:59 | sean-k-mooney | i would guess the nova bug for lossing message is still open | |
| 17:07:11 | sean-k-mooney | its possible that is what eandersson hit | |
| 17:10:07 | melwitt | I don't recall a nova bug for this one | |
| 17:10:18 | melwitt | in lp | |
| 17:10:58 | sean-k-mooney | i think there was one but my seach foo is failing | |
| 17:11:10 | sean-k-mooney | is there a way to search for bugs you commented on? | |
| 17:12:01 | sean-k-mooney | oh there is | |
| 17:12:21 | melwitt | here's the IRC convo about it from the same day we commented on the oslo.messaging review http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-06-06.log.html#t2019-06-06T22:47:09 | |
| 17:13:04 | sean-k-mooney | oh thats clever i would not have thought of that. i rememerned it was mnaser that hit it | |
| 17:15:13 | melwitt | rest of the convo is here and I don't see any nova lp bug mentioned other than an old one that got no additional info on it | |
| 17:15:15 | melwitt | http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-06-07.log.html | |
| 17:15:19 | sean-k-mooney | ok so there wasnt a specifc nova bug but there were 3 related bugs | |
| 17:15:23 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1794706 | |
| 17:15:23 | openstack | Launchpad bug 1794706 in OpenStack Compute (nova) "The instance left stuck when oslo.messaging raised MessageDeliveryFailure exception" [Undecided,Expired] | |
| 17:15:27 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1437955 | |
| 17:15:27 | openstack | Launchpad bug 1437955 in oslo.messaging "RPC calls and responses do not use the mandatory flag (AMQP)" [Wishlist,Confirmed] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 17:15:31 | melwitt | yeah | |
| 17:15:32 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1661510 | |
| 17:15:32 | openstack | Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio) | |
| 17:16:12 | sean-k-mooney | eandersson: so i think you were hitting the same issue as mnaser | |
| 17:17:02 | sean-k-mooney | melwitt: thanks :) | |
| 17:17:37 | melwitt | :) | |
| 17:18:17 | sean-k-mooney | shoudl i un expire https://bugs.launchpad.net/nova/+bug/1794706 by the way | |
| 17:18:17 | openstack | Launchpad bug 1794706 in OpenStack Compute (nova) "The instance left stuck when oslo.messaging raised MessageDeliveryFailure exception" [Undecided,Expired] | |
| 17:18:56 | sean-k-mooney | i guess there is no point we dont have the info we need | |
| 17:22:09 | eandersson | Yea that sounds like the exact issue | |
| 17:23:30 | eandersson | btw another major issue with this is that unless you know what you are doing it's very difficult to identify the bad compute | |
| 17:23:42 | melwitt | sean-k-mooney: yeah, I'd say don't bother, it's from 2017 and not enough info to move forward, unless I've missed something | |
| 17:23:56 | melwitt | oh, nvm 2018. I can't read | |
| 17:27:23 | melwitt | sean-k-mooney: if you wanted to unexpire and add more info to it based on eandersson experience (if it's the same thing) then I think that makes sense | |
| 17:27:27 | sean-k-mooney | eandersson: yes. would you feel comfortable writing this up as a bug | |
| 17:27:51 | sean-k-mooney | melwitt: its not exactly the same thing as the old nova bug | |
| 17:27:57 | sean-k-mooney | but i think its the same oslo bug | |
| 17:28:01 | melwitt | ack | |
| 17:28:06 | sean-k-mooney | which is what mnaser was hitting | |
| 17:28:11 | eandersson | I am pretty sure what happened here was that we had a rabbitmq network partition weeks ago | |
| 17:28:20 | eandersson | and that partition somehow damaged the queue | |
| 17:28:32 | sean-k-mooney | yep | |
| 17:28:43 | eandersson | but only from openstack perspective, because I could consume the queue using the ui etc. | |
| 17:28:43 | sean-k-mooney | basically if the que gets deleted | |
| 17:28:52 | sean-k-mooney | and you dont restart the nova-compute agent | |
| 17:28:57 | sean-k-mooney | then it wont recreate it | |
| 17:28:58 | eandersson | and conductor | |
| 17:28:58 | eandersson | yea | |
| 17:29:10 | eandersson | the weird thing is that it existed in a "healthy" state with the bindings etc | |
| 17:29:20 | melwitt | and if we leverage the earlier mentioned oslo.messaging change, we can make it recover in nova? | |
| 17:29:51 | eandersson | If I can find another bad queue I can test | |
| 17:29:57 | sean-k-mooney | i think so | |
| 17:30:05 | melwitt | kewl | |
| 17:30:40 | sean-k-mooney | i think either the condocot or compute node would get and excpetion if tehy tried to do a topic send to the queue and that would allwo use to fix it by recreating the queues | |
| 17:30:50 | openstackgerrit | Merged openstack/nova stable/queens: lxc: make use of filter python3 compatible https://review.opendev.org/676500 | |
| 17:31:02 | openstackgerrit | Merged openstack/nova master: Make sure tox install requirements.txt with upper-constraints https://review.opendev.org/689152 | |
| 17:31:24 | sean-k-mooney | melwitt: i would need to fully re read the oslo feature but they had a recovery mechanium in mind | |