Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-30
15:12:38 mriedem since you didn't mention where the deadlock is occurring, it's hard to say
15:12:46 mriedem there was also https://github.com/openstack/nova/commit/e001b0538a9d413f864d39584c2aea0b9e200e4b#diff-a040eef16b56cc7862fb2a334c757dea but that was fixed in pike
15:13:27 wangy The db deadlock was caused by "INSERT INTO allocations "
15:14:28 fghaas mriedem, kashyap: all good points, thanks — will push update momentarily
15:15:01 wangy mriedem: thanks,I had added retry decorator, it did not help
15:15:52 mriedem wangy: seems that would be covered with https://github.com/openstack/nova/commit/e001b0538a9d413f864d39584c2aea0b9e200e4b#diff-a040eef16b56cc7862fb2a334c757dea
15:16:42 mriedem do you see "Performing DB retry for function" in the placement api logs?
15:16:50 mriedem if so, that means it's retrying on deadlock
15:17:03 mriedem if you see "DB exceeded retry limit" that means it hit the retry limit
15:17:24 mriedem and probably means you need more resources / more placement workers in your env
15:18:10 wangy mriedem: yes, I see the logs, thanks for your opinion, I will try it.
15:19:33 openstackgerrit Theodoros Tsioutsias proposed openstack/nova-specs master: Enable rebuild for instances in cell0 https://review.openstack.org/554218
15:21:40 Shilpa mriedem: HI.
15:21:56 Shilpa mriedem:Can you please help me to understand in case of nova evacuate, why we are restoing allocations at source?
15:22:34 Shilpa is it for rollback?
15:23:04 Shilpa or in case of evacuate, in case of success, how allocaions at source are cleared off? Is any process set for it?
15:23:48 mriedem i'm not sure what you mean by "restoring allocations at source"
15:24:16 mriedem if the evacuated source nova-compute service starts up again and instances have been successfully migrated from it, the allocations are cleaned up for the source node during that startup
15:24:46 mriedem here https://github.com/openstack/nova/blob/d1e38f06bb5282dd9bf81d979926a0ebbfd93caa/nova/compute/manager.py#L712
15:25:43 mriedem if the evacuation fails, the source node allocations should still exist in placement
15:26:19 Shilpa mriedem: yes in case of failure, observe source node allocations are still there in placement
15:26:38 mriedem if the evacuation fails, the destination node allocations, which are created by the scheduler, should be cleaned up here https://github.com/openstack/nova/blob/d1e38f06bb5282dd9bf81d979926a0ebbfd93caa/nova/compute/manager.py#L3044 or here https://github.com/openstack/nova/blob/d1e38f06bb5282dd9bf81d979926a0ebbfd93caa/nova/compute/manager.py#L3058
15:26:55 Shilpa mriedem: will check for evacuated source nova-compute service starts up again, and what happens there
15:26:59 gibi mriedem: thanks for the review on https://review.openstack.org/#/c/567267/ I have to get back to these patches next week. Now I'm in a heavy preparation for our bandwidth demo for the summit
15:27:30 openstackgerrit sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034
15:28:04 mriedem gibi: ok, the changes don't seem very difficult for either change, would be ok with me just addressing my comments if you don't get to them first? so we can keep moving that series?
15:28:16 mriedem *would you be ok
15:29:23 mriedem tssurya: i'm not sure how caught up your are on the changes dansmith made to the handling a down cell series, but i've got a slew of comments in the majority of the bottom changes - are you going to address those this week? if not, i might start working on that as well since i think dansmith is busy
15:30:30 openstackgerrit sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034
15:30:43 Shilpa mriedem: checked that if evacuated source nova-compute service starts up again and instances have been successfully migrated from it, the allocations are cleaned up for the source node during that startup
15:31:03 Shilpa mriedem: thanks for this explaination
15:32:53 mriedem Shilpa: yw
15:33:04 bauzas mriedem: oops sorry I meant cast_as_calls
15:33:22 bauzas this ^ fixed my tz issue for the functional tests
15:33:26 sean-k-mooney Shilpa: note that technicall the instanaces are not migrated. they are recreaed during evacuate
15:33:36 mriedem bauzas: you mean the CastAsCallFixture right?
15:33:42 bauzas yup
15:33:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop request spec migration code https://review.openstack.org/614265
15:33:46 mriedem either way, i don't want to use that fixture if it can be helped
15:33:47 sean-k-mooney Shilpa: but if the source comes back up it will fix the allcoations
15:33:51 mriedem i'm also not sure why that would make a difference
15:34:08 mriedem i suppose because it stubs out the prepare method in the rpc client so we don't hit the validation in the fake rpc driver
15:34:21 bauzas mriedem: I didn't really had an issue when testing live, so I didn't went further down
15:34:43 dansmith mriedem: just saw your comment on the cells fixture thing
15:34:43 bauzas mriedem: yeah what you say rings a bell to me
15:34:45 bauzas but that's one
15:34:49 bauzas old*
15:34:52 Shilpa sean-k-mooney: thanks, understood
15:34:52 mriedem i know we have functional tests that don't use the CastAsCallFixture which still pass a request spec to select_destinations and build_and_run_instance
15:34:54 mriedem so i'm still confused
15:35:00 dansmith mriedem: if you hit a problem with that, there should be a runtimeerror raised
15:35:31 bauzas mriedem: FWIW, I'm disecting some internal bug for a customer that sounds pretty similar to https://bugs.launchpad.net/nova/+bug/1600251
15:35:32 openstack Launchpad bug 1600251 in OpenStack Compute (nova) "live migration does not honor server group policy" [High,Fix released]
15:36:00 dansmith mriedem: like, it re-checks with the lock held, so if you don't see that, it should be the same state as it was before that change
15:36:04 bauzas mriedem: mmm, good point about the other functional tests
15:36:16 dansmith mriedem: now, entirely possible that this shook up a race that we already had and were winning though
15:36:31 sean-k-mooney bauzas: spefically is that related to migratin multiple instnace in the server group conurrently?
15:36:41 bauzas sean-k-mooney: yup
15:36:51 sean-k-mooney bauzas: ya that is totally undefiened
15:36:58 mriedem bauzas: well that race has never been fixed
15:37:11 mriedem see my comments in the bug about what starlingx did to try and mitigate it
15:37:18 bauzas yup I saw them
15:37:46 mriedem dansmith: yeah maybe, i haven't dug too deep
15:40:53 sean-k-mooney dansmith: well exclutding nfv instnaces we sould have only won the race reliably if there was no race e.g. no concurrent ligrations. even in the singel sechduler case i dont belive we record the instance is on a node untill it boots so there would always be the race between the schduler selecting the node for the second insnance and the first instance booting right
15:42:00 dansmith sean-k-mooney: um, what?
15:42:29 dansmith sean-k-mooney: we talking about the same bug/
15:42:35 tssurya mriedem: I am catching up on them and addressing them
15:42:44 tssurya I will be working on them this week
15:43:01 tssurya my major concern being https://review.openstack.org/#/c/567785/29/nova/compute/api.py@2658
15:43:06 sean-k-mooney sorry :) current migration of instance in a server group can race on instance boot even with one scuderler becasue of when we update the instance members https://bugs.launchpad.net/nova/+bug/1600251
15:43:07 openstack Launchpad bug 1600251 in OpenStack Compute (nova) "live migration does not honor server group policy" [High,Fix released]
15:43:23 sean-k-mooney i was talking about the one bauzas linked so maybe not
15:43:29 tssurya (just back from vacation so excuse me if I am missing something obvious)
15:43:30 gibi mriedem: if you have time to work with those patches then please go ahead. and thank you
15:45:10 melwitt efried: looks like johnthetubaguy already got it :)
15:49:22 mriedem tssurya: i'll reply in the review
15:49:33 tssurya thanks
15:51:54 melwitt kashyap: hey, your review on this tcp-based console log patch would be appreciated https://review.openstack.org/575735
15:52:19 kashyap melwitt: Morning, will look
15:52:24 melwitt thanks
15:56:07 openstackgerrit Florian Haas proposed openstack/nova master: Explain cpu_model_extra_flags and nested guest support https://review.openstack.org/609788
15:57:12 cfriesen following a discussion with mriedem late last Friday, the HPET spec has been updated with something much simpler. (Turns out no trait is needed after all.) https://review.openstack.org/#/c/607989
15:58:26 openstackgerrit Florian Haas proposed openstack/nova stable/rocky: Explain cpu_model_extra_flags and nested guest support https://review.openstack.org/609789
15:59:45 openstackgerrit Stephen Finucane proposed openstack/nova master: Fail to live migration if instance has a NUMA topology https://review.openstack.org/611088
16:00:09 openstackgerrit Florian Haas proposed openstack/nova stable/queens: Explain cpu_model_extra_flags and nested guest support https://review.openstack.org/609790
16:04:32 openstackgerrit Florian Haas proposed openstack/nova master: Explain cpu_model_extra_flags and nested guest support https://review.openstack.org/609788
16:05:22 johnthetubaguy melwitt: there is a first time for everything :)
16:05:47 melwitt heh :P
16:06:10 openstackgerrit Florian Haas proposed openstack/nova stable/rocky: Explain cpu_model_extra_flags and nested guest support https://review.openstack.org/609789
16:07:47 openstackgerrit Florian Haas proposed openstack/nova stable/queens: Explain cpu_model_extra_flags and nested guest support https://review.openstack.org/609790
16:09:44 efried melwitt, johnthetubaguy: Thanks!
16:10:03 openstackgerrit sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034
16:34:36 melwitt johnthetubaguy: hey, thanks for the review! this is the corresponding reno update if you wouldn't mind https://review.openstack.org/613174
16:34:57 johnthetubaguy melwitt: ah, missed that, looking
16:35:17 melwitt that one has to go direct to stable, in order to update a already release reno
16:35:25 melwitt thanks
16:36:55 melwitt oh, dangit I need to rebase
16:37:09 melwitt [the master change]
16:38:11 melwitt I used to have a master branch reno update on top, then learned it has to go direct to stable, so I abandoned the one on master, but that's preventing the approved bottom change from going to the gate
16:39:34 gaudenz Cloud someone have a look at https://review.openstack.org/#/c/523006/ ? This fixes a bug that has been around for a few releases and I made an effort to rebase it just a few days ago. Now all tests pass again. The bug is about resource usage calculations flapping between what nova calcualtes and what libvirt reports.

Earlier   Later