| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-21 | |||
| 09:25:20 | kashyap | Let me go do that | |
| 09:26:01 | bauzas | gibi: at least we can ask the reporter to tell us whether nova still sees it => Incomplete | |
| 09:26:35 | lyarwood | kashyap: make my q35 change depend on your tempest change | |
| 09:27:02 | kashyap | lyarwood: Yeah. Not sure what I'm thinking | |
| 09:27:08 | kashyap | Doing ... | |
| 09:27:13 | gibi | bauzas: OK, I was not able to formulate any valid question to the reporter. I glad you could | |
| 09:27:23 | bauzas | gibi: I'm just doing it | |
| 09:30:13 | kashyap | lyarwood: Can I remove the Depends-On in your 'q35' CI change against the 'blkid -L' change? | |
| 09:31:30 | kashyap | lyarwood: I'll remove it for now, so as to test one thing at a time -- okay? | |
| 09:32:37 | lyarwood | kashyap: well it's seperate | |
| 09:32:50 | lyarwood | kashyap: ah wait no it isn't | |
| 09:32:57 | lyarwood | kashyap: it's in openstack/tempest so yeah feel free to | |
| 09:33:22 | lyarwood | kashyap: btw I'm planing on writing a spec for the q35 default change next week | |
| 09:33:25 | kashyap | Okido; I'll add a note in the commit with a link to tit. | |
| 09:33:53 | kashyap | lyarwood: FWIW, I already have it in place somewhere | |
| 09:34:01 | kashyap | One is this BP: https://blueprints.launchpad.net/nova/+spec/gracefully-handle-qemu-machine-types | |
| 09:34:06 | kashyap | And a spec should also be around ... | |
| 09:35:04 | kashyap | lyarwood: Here's the spec started 8 months ago: https://review.opendev.org/#/c/631154/ | |
| 09:35:27 | lyarwood | kk I'll repropose that for V | |
| 09:37:07 | kashyap | lyarwood: I was meaning to send a note to the upstream list as a heads-up. | |
| 09:37:45 | kashyap | (I have a draft e-mail, will send it sometime this week. Giving a heads-up to prepare ... as this is a virtual 'motherboard' change) | |
| 09:39:39 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701 | |
| 09:41:38 | lyarwood | kashyap: lets get the spec reproposed, updated and on the agenda for PTG first | |
| 09:42:29 | lyarwood | kashyap: I'm not going to be around for PTG btw would you be okay leading any sessions/discussions on the ML? | |
| 09:42:53 | bauzas | gibi: FYI I think I found the problem but before triaging it to Confirmed, I need to ask the reporter to tell us whether I'm right https://bugs.launchpad.net/nova/+bug/1860312/comments/1 | |
| 09:42:53 | openstack | Launchpad bug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,Incomplete] | |
| 09:42:58 | kashyap | lyarwood: Yeah, I was intending to that. As I've been delaying this work for various reasons | |
| 09:43:56 | gibi | bauzas: cool | |
| 09:44:08 | gibi | bauzas: also I left some comments on the prelude. good job! | |
| 09:44:31 | bauzas | gibi: ack | |
| 09:45:05 | gibi | stephenfin: could you re-review stable device rescue doc patch? https://review.opendev.org/#/c/700837 | |
| 09:45:09 | bauzas | I wish I could easily find bug duplicates with Launchpad, because I'm pretty sure that bug 1860312 is not the only one for the same issue | |
| 09:45:09 | openstack | bug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,Incomplete] https://launchpad.net/bugs/1860312 | |
| 09:45:17 | stephenfin | sure, just finished gmann doc change | |
| 09:45:21 | stephenfin | *gmann's | |
| 09:45:32 | gibi | stephenfin: that would have been my other request :) | |
| 09:46:54 | bauzas | gibi: bingo : https://bugs.launchpad.net/nova/+bug/1853009 | |
| 09:46:54 | openstack | Launchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard) | |
| 09:47:53 | mgoddard | bauzas, gibi: would love some reviews on those patches :) | |
| 09:48:10 | bauzas | mgoddard: absolutely, will do | |
| 09:48:13 | mgoddard | I think some were failing unit tests, I can tidy up if someone plans to look | |
| 09:48:28 | bauzas | mgoddard: how many times ironic can rebalance ? | |
| 09:48:46 | bauzas | because I think that can be a problem for a shit ton of deployers using ironic | |
| 09:48:55 | mgoddard | bauzas: advance warning: prepare a cup of coffee, or three. Nasty race conditions ahead | |
| 09:49:08 | mgoddard | bauzas: unlimited rebalancing | |
| 09:49:15 | stephenfin | lyarwood, gibi: done https://review.opendev.org/#/c/700837/ | |
| 09:49:15 | bauzas | in particular when it goes on top on something abstract API like k8s... | |
| 09:49:52 | mgoddard | bauzas: the hash ring just consists of the active nova computes | |
| 09:49:54 | bauzas | mgoddard: just tbc, ironic does rebalance if the things below it are shuffled ? | |
| 09:50:12 | gibi | bauzas: ack about the duplicates | |
| 09:50:16 | gibi | stephenfin: thanks | |
| 09:50:18 | bauzas | in our case, because pods changed or whatever, right? | |
| 09:50:20 | lyarwood | stephenfin: ah thanks, ./me respins | |
| 09:50:38 | mgoddard | bauzas: there are two hash rings - one in nova, one in ironic conductor. You only need to care about the one in nova | |
| 09:50:46 | bauzas | mgoddard: you do confirm that https://bugs.launchpad.net/nova/+bug/1860312 sounds dup of https://bugs.launchpad.net/nova/+bug/1853009 ? | |
| 09:50:46 | openstack | Launchpad bug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,Incomplete] | |
| 09:50:47 | openstack | Launchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard) | |
| 09:51:51 | mgoddard | bauzas: if it's sitting on top of a wobbly k8s jelly, with containers coming and going, it's quite likely :) will look in more detail | |
| 09:51:53 | bauzas | mgoddard: I'd dare say there are two hard problems in CS | |
| 09:52:13 | bauzas | hash rings come with the first hard problme | |
| 09:52:56 | bauzas | mgoddard: that's exactly what the reporter says " | |
| 09:52:56 | bauzas | I deployed openstack with openstack-helm on kubernetes.When one of the nova-compute service(driver=ironic replica of the deployment is 1) breakdown.It may be scheduled to another node by kubernetes.When I try to delete the old compute service(status down), it failed." | |
| 09:53:39 | mgoddard | bauzas: although in this case I think it's a different problem. They're trying to delete the compute service, not the compute node | |
| 09:53:58 | mgoddard | bauzas: I'd guess it's because they're using the service ID rather than the UUID | |
| 09:54:02 | mgoddard | ID no longer seems to work | |
| 09:54:16 | mgoddard | need to specify an API version to get the UUID in the list | |
| 09:55:20 | bauzas | mgoddard: it's a side effect | |
| 09:55:43 | bauzas | mgoddard: since the virt driver stopped reporting the node, then the RT deleted the compute node record | |
| 09:55:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: docs: Add stable device rescue docs https://review.opendev.org/700837 | |
| 09:56:49 | lyarwood | stephenfin: ^ thanks btw | |
| 09:57:05 | mgoddard | bauzas: but the service should still exist, even without a node | |
| 09:57:31 | bauzas | mgoddard: we're making different assumptions in different pieces of code | |
| 09:57:55 | mgoddard | bauzas: I haven't seen that happen before | |
| 09:58:14 | bauzas | mgoddard: in the RT, we assume that it's reasonable to not have a CN record when the virt driver tells you no about it | |
| 09:58:35 | bauzas | mgoddard: but on the other hand, we assume that compute nodes exist if you delete a compute service | |
| 10:00:22 | mgoddard | bauzas: if that is true, should be easy to reproduce? | |
| 10:00:37 | mgoddard | and probably easy to test my hypothesis also | |
| 10:03:30 | bauzas | mgoddard: I've asked the reporter to come back with details | |
| 10:04:21 | openstackgerrit | Merged openstack/nova master: Use oslo policy flag to disable default change warning instead of all https://review.opendev.org/717884 | |
| 10:04:26 | bauzas | I could have some OSP crazy env that would help me to verify this hypothesis but to be frank I don't have the energy for it :) | |
| 10:05:05 | bauzas | mgoddard: like I said in my last comment of bug 1860312, I'd rather spend my energy on reviewing your change | |
| 10:05:05 | openstack | bug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,Incomplete] https://launchpad.net/bugs/1860312 | |
| 10:08:36 | bauzas | mgoddard: actually, the bug is pretty nasty and has a lot of other related bugs, do you think you'd be able to give it a good shape soon ? | |
| 10:08:54 | bauzas | to give *the change a good shape soon ? | |
| 10:10:38 | bauzas | gibi: FYI, ^ | |
| 10:11:29 | bauzas | we're too close to RC1 to consider those bugs as rc-potential but they're definitely worth backportable down the road to all our stable branches | |
| 10:12:08 | bauzas | actually, mriedem thought the same https://bugs.launchpad.net/nova/+bug/1841481 | |
| 10:12:08 | openstack | Launchpad bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 10:12:41 | gibi | bauzas: ack, I agree that we should not try to blow up RC1 with this fix. | |
| 10:12:45 | bauzas | mgoddard: sorry for interupting you again, but we have around 4 open bugs for probably the same issue(s) or related | |
| 10:12:59 | bauzas | mgoddard: could we consider https://bugs.launchpad.net/nova/+bug/1853009 as the primary one ? | |
| 10:13:00 | openstack | Launchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] - Assigned to Mark Goddard (mgoddard) | |
| 10:14:16 | mgoddard | bauzas: I'll get the patches into shape today | |
| 10:15:15 | bauzas | mgoddard: as I said, we're too short in time to deliver them for RC1, but I think this should be addressed as quicky as possible after the victoria branch | |
| 10:15:48 | mgoddard | bauzas: ack | |
| 10:22:05 | bauzas | mgoddard: fwiw, starting to review https://review.opendev.org/#/c/695012/4/nova/tests/functional/regressions/test_bug_1853009.py | |
| 10:23:04 | mgoddard | bauzas: cool. I don't expect the code to change significantly, just need to adapt to upstream changes since proposed | |
| 10:39:23 | gibi | bauzas: what do you think about https://bugs.launchpad.net/nova/+bug/1872385 it is currently marked as rc potential but it the fix is not ready. I can take the fix over this afternoon and make it complete or we can drop the rc potential tag as you also noted that this is not a clear regression for Ussuri as we have child RPs for couple of releases now | |
| 10:39:23 | openstack | Launchpad bug 1872385 in OpenStack Compute (nova) "delete sub resource provider in tree" [High,In progress] - Assigned to Wenping Song (wenping1) | |
| 10:47:09 | sean-k-mooney | gibi: i dont have it deploy but i have worked with dvd on https://bugs.launchpad.net/nova/+bug/1851545 downstream https://bugzilla.redhat.com/show_bug.cgi?id=1767797 | |