Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-11
16:22:36 mriedem oh
16:23:39 mriedem i seem to remember some tempest bugs we were trying to fix around the time of this change https://review.openstack.org/#/c/427782/
16:25:43 mdbooth mriedem: Any chance you could look at https://review.openstack.org/#/c/510106/ ? It's a reapproval. You asked for some changes during Newton, which I've added to this version of the spec.
16:26:08 mriedem not right now
16:26:12 mdbooth lyarwood's stable virt
16:26:28 mdbooth mriedem: Ok. Is it likely to make next week?
16:26:38 mriedem idk
16:26:44 mriedem lots of people can review specs
16:26:53 mriedem i've got my own to write and update as well
16:27:11 mriedem kind of concerned with a bunch of people saying they are busted on ocata atm
16:27:12 mdbooth mriedem: np, just pinging you specifically as you've been involved with it previously.
16:32:07 mriedem dansmith: wonder if these people are running with uwsgi? :)
16:32:13 mriedem remember that defeated the service version check
16:32:15 mriedem for tripleo
16:32:17 dansmith mriedem: ocata so no right?
16:32:25 dansmith oh
16:32:26 mriedem but they were hitting something similar,
16:32:29 mriedem unti lwe told them not to do that
16:32:48 dansmith hmm
16:33:11 openstackgerrit Ildiko Vancsa proposed openstack/nova master: Fix connection info refresh for reboot https://review.openstack.org/511209
16:33:51 dansmith mriedem: see comment 9
16:33:57 dansmith mriedem: makes reference to having everything upgraded
16:35:45 mriedem yeah i think there are two different groups of people with different issues
16:35:55 mriedem once is talking about having old computes, which shouldn't have anything to do with looking up the instnace
16:36:09 mriedem i think the others are talking about a race to lookup the instance while it's building
16:36:21 mriedem which is why they added a sleep in there i guess and said it's working
16:36:37 dansmith well, one unupdated api will wreck it right?
16:36:50 mriedem b/c of the api version check?
16:36:53 dansmith yeah
16:37:02 dansmith it'll go straight to the cell db and not look at the BR
16:37:35 mriedem yeah
16:37:43 mriedem but, someone said they have a full new install
16:37:52 mriedem and that was the person that put the time.sleep(1) workaround in
16:38:29 dansmith all they need is one api service record that reported zero for the version
16:38:49 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Network bandwidth resource provider https://review.openstack.org/502306
16:39:37 dansmith and they did it before that check
16:39:43 dansmith the sleep I mean
16:41:30 openstackgerrit Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Network bandwidth resource provider https://review.openstack.org/502306
16:48:05 mriedem assuming their api services are all updated, i don't see how they could hit this really
16:48:34 mriedem we delete the build request after the instance mapping is updated,
16:48:40 mriedem we update the instance mapping after the instance is created in a cell
16:49:02 openstackgerrit melanie witt proposed openstack/nova stable/pike: Target context for build notification in conductor https://review.openstack.org/511292
16:49:04 mriedem if the instance mapping doesn't have the cell mapping in it, we get the instance from the build request - and if that's gone by the time we look, we check the instance mapping again
16:49:27 dansmith mriedem: aye
16:49:43 mriedem gdi this is like 2 days i've spent all day chasing weird bugs
17:21:15 mriedem melwitt: does https://review.openstack.org/#/c/511292/ also need to go to ocata?
17:22:43 melwitt hm, lemme check. I had thought not but maybe this code was from ocata
17:22:44 mriedem looks like yes https://github.com/openstack/nova/blob/stable/ocata/nova/conductor/manager.py#L921
17:23:11 mriedem it also looks like ocata is not targeting the instance action in the cell, so that's getting created in cell0
17:23:22 melwitt yeah, was about to say that too
17:23:54 openstackgerrit Dan Smith proposed openstack/nova master: Revert allocations by migration uuid https://review.openstack.org/498949
17:23:55 openstackgerrit Dan Smith proposed openstack/nova master: Refactor resource tracker to account for migration allocations https://review.openstack.org/506419
17:23:55 openstackgerrit Dan Smith proposed openstack/nova master: Make migration uuid hold allocations for migrating instances https://review.openstack.org/506420
17:23:56 openstackgerrit Dan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation https://review.openstack.org/507638
17:23:56 openstackgerrit Dan Smith proposed openstack/nova master: Split get_allocations_for_instance() into useful bits https://review.openstack.org/511306
17:24:10 mriedem https://github.com/openstack/nova/commit/ca51db7e2db72cb774c94fb039509f2b4e6c94ca
17:24:19 dansmith mriedem: ^ updated to determine the source compute from allocations instead of trying to look it up
17:24:37 mriedem melwitt: https://github.com/openstack/nova/commit/ca51db7e2db72cb774c94fb039509f2b4e6c94ca fixed the instance action thing in pike, we'll want to backport that as well...
17:24:41 mriedem didn't have a bug o nit
17:24:51 melwitt ah, crap
17:25:17 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Make conductor create InstanceAction in the proper cell https://review.openstack.org/511307
17:25:20 mriedem thar she blar
17:25:27 melwitt k, cool
17:25:50 melwitt I should stack the other backport on that I think
17:26:01 mriedem yup
17:26:15 mriedem dansmith: ok
17:26:18 melwitt to increase the chance of clean backport. k
17:26:22 mriedem dansmith: +2 on mel's pike backport https://review.openstack.org/#/c/511292/
17:26:52 mriedem i need food in my belly
17:32:12 jaypipes dansmith: I think zuul might be lying about the merge failure messages.
17:32:29 dansmith jaypipes: I don't think they've actually started the cutover yet
17:32:31 dansmith which means, yes
17:34:12 melwitt it's expected to emit false merge conflicts prior to cutover? I guess I am missing something obvious
17:34:29 dansmith it's been doing that all week
17:34:38 melwitt I know
17:35:48 dansmith I'm just saying, it's been doing it all week, and the changes they have planned today haven't started yet AFAIK, so I think we'd expect the same behavior
17:36:20 dansmith I would assume that whatever has been giving zuul merge conflict heartburn over the last week is factored into the switchover
17:36:35 melwitt I had been thinking they were fixing zuul v3 before the cutover so maybe those would get fixed prior to the cutover
17:37:10 melwitt yeah. I guess I should ask
17:37:40 dansmith unless the failure is because they're not having it do the post-merge conflict checks because of resources or something
17:37:49 dansmith since it's highly constrained at the moment, and not doing the actual merges
17:37:55 openstackgerrit Jay Pipes proposed openstack/nova master: placement: add nested resource providers https://review.openstack.org/377138
17:37:56 openstackgerrit Jay Pipes proposed openstack/nova master: placement: allow filter providers in tree https://review.openstack.org/377215
17:37:56 openstackgerrit Jay Pipes proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807
17:37:57 openstackgerrit Jay Pipes proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693
17:38:11 dansmith I'm not sure what jaypipes' point was I guess
17:38:15 jaypipes efried: k, updated the n-r-p spec to address your comments ^^
17:38:28 efried Thanks jaypipes
17:38:41 jaypipes dansmith: generally, you can assume I am pointless.
17:38:51 efried dansmith jaypipes They figured out why the bogus merge conflicts were happening.
17:38:55 efried But no, they haven't cut over yet.
17:39:17 efried http://eavesdrop.openstack.org/irclogs/%23openstack-infra-incident/%23openstack-infra-incident.2017-10-11.log in case you're interested.
17:39:54 melwitt indeed they know about it in #infra and they're working on it
17:40:12 dansmith so yeah, like I said, it's getting old data because it's not doing the actual merges
17:40:46 melwitt okay. I didn't know how it worked
17:41:27 jaypipes efried, dansmith: k, thx for the info.
17:41:58 openstackgerrit Dan Smith proposed openstack/nova master: Revert allocations by migration uuid https://review.openstack.org/498949
17:41:59 openstackgerrit Dan Smith proposed openstack/nova master: Refactor resource tracker to account for migration allocations https://review.openstack.org/506419
17:41:59 openstackgerrit Dan Smith proposed openstack/nova master: Make migration uuid hold allocations for migrating instances https://review.openstack.org/506420
17:42:00 openstackgerrit Dan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation https://review.openstack.org/507638

Earlier   Later