Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-13
13:49:53 tssurya but like you said as least for ironic the client assumes everything is done correctly on server side
14:03:22 redkrieg lyarwood: thanks for getting back to me. if there's anything I can do to help with stable device rescue, please let me know. it would solve some big pain points for me and my clients.
14:07:38 efried slaweq: I've seen https://bugs.launchpad.net/nova/+bug/1839961 quite a bit as well. mriedem is our resident bugmeister, though, perhaps he has insight.
14:07:38 openstack Launchpad bug 1669468 in devstack "duplicate for #1839961 tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc fails intermittently in neutron multinode nv job" [Medium,Fix released] - Assigned to melanie witt (melwitt)
14:08:00 slaweq efried: thx, yes, mriedem already marked it as duplicate of other bug
14:08:03 slaweq sorry for noise than :)
14:08:09 mriedem the devstack patch was merged yesterday
14:08:18 slaweq and thx for fix
14:08:24 mriedem http://status.openstack.org/elastic-recheck/#1669468
14:08:33 mriedem if it's still hitting we might have other problems
14:08:42 mriedem https://review.opendev.org/#/c/675721/
14:09:29 efried now the list is down to 167
14:16:42 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140
14:17:21 mriedem tssurya: when are you on vacation?
14:26:31 Tianhao_Hu About this issue, there are files left after cold migration every time. But the directory left is empty and has no effect on cold migration. I'm wondering that if we can think this is not a bug? Can anyone give me some help on this?
14:26:35 Tianhao_Hu https://bugs.launchpad.net/starlingx/+bug/1824858
14:26:35 openstack Launchpad bug 1824858 in StarlingX "nova instance remnant left behind after cold migration completes" [Low,Confirmed] - Assigned to hutianhao27 (hutianhao)
14:30:49 mriedem tssurya: dansmith: efried: https://review.opendev.org/#/c/645611/
14:32:05 mriedem i didn't compare to my last set of comments (PS7) because the diff is too large
14:32:22 mriedem tssurya: if you're going to be on vacation soon, like leaving today/tomorrow, then i can take this over
14:33:09 mriedem i think my main concern is whether or not operators can disable ironic sending the event until nova is completely upgraded
14:33:28 efried They can, there's a conf option on the ironic side.
14:33:48 efried https://review.opendev.org/#/c/664842/19/ironic/conf/nova.py
14:33:50 mriedem efried: you mean just to disalbe nova?
14:33:53 dansmith and unless ironic really cares, it shouldn't be an issue
14:34:02 mriedem ah send_power_notifications
14:34:06 mriedem yeah we should mentio nthat in the nova release note
14:34:10 dansmith I expect ironic is sending these to be informative, but not depending on the result
14:34:16 mriedem dansmith: that's not the issue,
14:34:50 mriedem the issue is if you're doing rolling upgrades of your computes, you've upgraded ironic before nova, then nova control plane, and you can be getting the events, the api sets the task_state on the instance and casts to an old compute that doesn't know how to handle this event and then your instance is stuck with a task_state that won't go back to None w/o admin intervention
14:35:20 dansmith okay yeah, good point
14:35:23 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP! tox: Add mypy target https://review.opendev.org/676208
14:35:23 openstackgerrit Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci' https://review.opendev.org/676209
14:35:24 openstackgerrit Stephen Finucane proposed openstack/nova master: zuul: Add 'mypy' job https://review.opendev.org/676210
14:35:33 dansmith even if we can disable the ironic side, the nova side needs to not do that
14:36:25 dansmith tbh, the api setting the task state has kinda rubbed me the wrong way the whole time anyway,
14:36:39 dansmith as it seems like we should just forward this down to the compute and let it do its thing if needed
14:36:57 mriedem i mentioned in ealier review that the API could check the compute service version, but because volume-extended was merged w/o doing that and has the same issue, it was going to slide here as well
14:37:19 dansmith setting task state is to help with races to take action on events, and if we can't do the power state short-circuit because something else has started, it's fine to fail/log late
14:37:32 stephenfin mriedem: Question on https://review.opendev.org/#/c/631581/44/nova/conductor/tasks/cross_cell_migrate.py
14:37:57 mriedem dansmith: that was another thing i pointed out, the API code isn't handling UnexpectedTaskStateError if we lose a race so the API would 500 and fail for all events
14:38:10 mriedem *all events in that request
14:39:02 dansmith mriedem: yeah, so I think I'd use this as an argument to not make this something more complex on the api side
14:41:37 mriedem stephenfin: replied
14:45:05 mriedem stephenfin: i realized late that the cinder 3.44 check in there can be removed now https://review.opendev.org/#/c/649759/
14:45:15 mriedem i wrote the cross-cell patch before we bumped the min required cinder,
14:45:18 mriedem so i'll remove that in a follow up
14:45:33 stephenfin I also missed that. Yeah, follow-up would be good (y)
15:09:56 dansmith mriedem: I commented on tssurya's patch about the api task_state involvement, and also thought of another problem with that
15:26:18 mriedem dansmith: i don't think the libvirt thing is an issue if you're compute is train+
15:26:37 dansmith no?
15:26:37 mriedem b/c the driver will raise NotImplementedError and @reverts_task_state will set the task_state to None
15:26:54 dansmith is that a call?
15:26:56 mriedem i commented on that in one of the unit tests - as in that scenario is not asserted
15:27:09 mriedem @reverts_task_state is a decorator on the compute managermethod
15:27:27 dansmith ah
15:28:30 mriedem so i think we're covered there, it's just not tested :(
15:28:47 dansmith okay I see, she's got it on the helper method
15:28:54 dansmith I was pretty sure it was not on the event handler function itself
15:32:44 mriedem dansmith: efried: fyi i'll be out on friday, going camping up north in the bOuNdArY wAtErS; if i'm not back next monday it's either because a bear ate me or i illegally emigrated to canada
15:33:02 efried ack
15:33:34 efried you're going with someone you can outrun, right?
15:33:47 mriedem until then i'll be planning on how to live w/o running water and on hot dogs for 6 straight meals
15:33:53 mriedem oh of course
15:51:10 openstackgerrit Merged openstack/nova stable/ocata: fix unshelve notification test instability https://review.opendev.org/674907
15:52:36 openstackgerrit Matt Riedemann proposed openstack/nova master: FUP to I30916d8d10d70ce25523fa4961007cedbdfe8ad7 https://review.opendev.org/676228
15:55:49 openstackgerrit Shilpa Devharakar proposed openstack/nova master: Nova object changes for forbidden aggregates request filter https://review.opendev.org/671072
15:55:49 openstackgerrit Shilpa Devharakar proposed openstack/nova master: DB API changes to get non-matching aggregates from metadata https://review.opendev.org/671074
15:57:20 openstackgerrit Shilpa Devharakar proposed openstack/nova master: Add a new request filter to isolate aggregates https://review.opendev.org/671075
15:58:08 openstackgerrit Shilpa Devharakar proposed openstack/nova master: Enable request filter isolate_aggregates https://review.opendev.org/667952
15:59:25 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.75 - Multiple API cleanup changes https://review.opendev.org/676230
16:08:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.opendev.org/631581
16:08:50 openstackgerrit Matt Riedemann proposed openstack/nova master: FUP to I4d181b44494f3b0b04537d5798537831c8fdf400 https://review.opendev.org/676231
16:09:21 mriedem wtf
16:09:40 mriedem oh sob
16:10:35 efried is mriedem wishing he had rebase=false in his .gitconfig?
16:10:48 mriedem no, i did git review -R, that's not the problem
16:11:25 mriedem the problem is i rebased HEAD~41, FUP to 41, git rebase --continue, FUP to 40, but that was not part of my main series so now it's diverged
16:11:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.opendev.org/631581
16:12:00 sean-k-mooney efried: im not sure how people end up with git reveiw rebasing there patches without them asking. i dont have rebase=false in my .gitconfig but i dont think i have ever had that proablem and dont then to use -R
16:13:06 sean-k-mooney at least in my normal workflow of "git review -d <gerrit number>; edit, git commit --amend, git review" that has never been a issue i have had
16:13:07 openstackgerrit Matt Riedemann proposed openstack/nova master: FUP to I4d181b44494f3b0b04537d5798537831c8fdf400 https://review.opendev.org/676231
16:13:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.opendev.org/633853
16:15:16 mriedem i'll just wait for the bottom things to merge and then do a proper rebase
16:16:04 efried mriedem: I don't think I understand the rolling upgrade with the ironic event thing.
16:16:43 efried If ironic sends the event to a nova that can't understand it, it'll just bounce, and the behavior will be as if we didn't have the feature, and the nova periodic will force the instance state like it does today -- right??
16:16:53 mriedem no
16:17:11 mriedem traditional upgrade sequence for nova and ironic is ironic first because the driver didn't do version discovery,
16:17:20 mriedem i think that's changed in recent releases but not sure how robust it is,
16:17:33 mriedem so you upgrade ironic, then the nova control plane, and you can have Stein computes for rolling upgrades, right?
16:17:42 mriedem so your nova api is train exposing 2.76
16:18:00 dansmith yup
16:18:08 efried ah, so this is really a coordination problem where we need to tolerate n-api > n-cpu
16:18:08 mriedem ironic default to send the events and if ironic is configured for talking to nova,
16:18:10 mriedem it will send the events
16:18:14 mriedem yes
16:18:24 efried it really has little to do with the ironic side sending events or not.
16:18:29 mriedem this is why the api does min compute service version checking in places
16:18:36 mriedem correct
16:18:38 dansmith correct, which is why I said the ironic tunable is not a solution

Earlier   Later