Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-02
15:21:52 lyarwood ;)
15:22:10 sean-k-mooney so the long rpc might fix it ya
15:22:14 mriedem it's a 60 second timeout by default with a blocking api - if the compute is slow or we're waiting on the instance uuid lock (maybe because a periodic is doing something with the instance at the time of attach), we could reasonably timeout
15:22:37 mriedem https://review.opendev.org/#/c/566696/ only goes back to rocky though, so you can't get the long_rpc_timeout fix to queens
15:22:43 mriedem so no use in saying that's the backportable fix
15:23:33 mriedem anyway, it just seems like a really small window where the fix in the api would handle it since the bdm create is the last thing to happen - if we timeout before that we still orphan the bdm record
15:23:54 mriedem and i know i'm contradicting my "what's the harm?" replies to mdbooth in the comments :)
15:24:07 mriedem the better longer term fix is the wip i posted, but that's not backportable
15:24:45 lyarwood yeah agreed on that part, we could backport long_rpc_timeout downstream *cough*
15:24:51 lyarwood but yeah
15:25:07 mriedem also your bug is definitely a duplicate of https://bugs.launchpad.net/nova/+bug/1425352
15:25:07 openstack Launchpad bug 1425352 in OpenStack Compute (nova) "A volume remains attached and cannot be detached after attaching it fails" [Low,Confirmed]
15:25:59 sean-k-mooney lyarwood: the issue would be the oslo.messaging min version bump
15:26:24 mriedem sean-k-mooney: correct we're not backporting that to queens-em upstream
15:27:14 lyarwood okay that's likely not an issue downstram
15:27:19 lyarwood downstream*
15:27:26 sean-k-mooney i dont know
15:27:37 sean-k-mooney i would not nessisarliy be comfrotable with that
15:27:46 sean-k-mooney but we could explore it as an option
15:28:07 sean-k-mooney it would depend on what version of oslo.messaging we are actully using downstream
15:28:16 lyarwood >= 5.29.0
15:28:29 lyarwood sean-k-mooney: when did this come in?
15:28:34 sean-k-mooney 6.3.0
15:29:08 lyarwood ah
15:29:11 lyarwood that would be an issue
15:29:17 sean-k-mooney https://github.com/openstack/requirements/blob/stable/queens/upper-constraints.txt#L145
15:29:23 sean-k-mooney so queens was capped a 5.35.6
15:29:31 lyarwood yeah guess what we use downstream
15:29:31 sean-k-mooney so 6.3.0 proably will break things
15:29:39 lyarwood okay that's dream is dead
15:29:42 lyarwood that*
15:30:14 sean-k-mooney is it higher then the upper-constraints for queens? cause i hate when we do that
15:30:42 lyarwood no it's not
15:30:52 sean-k-mooney cool
15:30:57 mriedem you could create your own workaround option as a timeout for this specific call that overrides the default rpc_response_timeout just for your queens-only change and drop it for long_rpc_timeout when you start using that for this call from upstream
15:32:44 sean-k-mooney ya is there an advantage to not useing the same config option
15:33:01 sean-k-mooney is it jsut to make it clear it applies to just this one call
15:37:24 mriedem lyarwood: i left some comments in the patch to try and summarize irc discussion and duplicated the bug to the existing one for the same issue. i'd hold off on writing a functional test until more people weigh in on what the correct direction is here given backports and such.
15:38:02 gibi sean-k-mooney: could you quickly respin https://review.opendev.org/#/c/695118/ to fix stephenfin nits in the conf doc and the release notes? The change looks good to me too but I guess you want to backpor it so a follow up won't help
15:38:58 sean-k-mooney yep i can do that in 5 mins thanks for taking a look
15:39:35 gibi sean-k-mooney: thanks. Ping me and I will +2
15:39:50 mriedem would be nice if any of the blizzard people would ack that change since they reported it
15:45:01 mriedem stephenfin: tbc i'm waiting on vmware ci results on https://review.opendev.org/#/c/696503/ before moving it forward
15:45:33 mriedem if that ci is busted (let's say we don't get results within the next 24 hours or something) then i'm cool with moving forward since if they can't maintain a working ci then we can't maintain their driver for them
15:45:43 stephenfin coolness
15:55:35 sean-k-mooney ... the release notes tox env takes forever to run
15:59:30 openstackgerrit sean mooney proposed openstack/nova master: add [libvirt]/max_queues config option https://review.opendev.org/695118
15:59:45 sean-k-mooney gibi: stephenfin ^ nits addressed no other changes
15:59:53 gibi sean-k-mooney: looking...
16:01:31 gibi eandersson: the bug is coming from blizzard ^^ could you check the bugfix from your perspective?
16:02:26 sean-k-mooney i will be filing and fixing a related bug for vhost-user seperatly but that should cover the case in the current bug if not let me know.
16:08:32 sean-k-mooney actully thinking about that i wonder if i should also add extra code to handel updating this on live migration?
16:11:12 sean-k-mooney i dont think we have code for the queue lenghts on live migration so i kind of feel a seperate patch to handel both queue lenght and max queues would make sense since that will need object changes and wont be backportable where as this simple fix is
16:13:15 sean-k-mooney gibi: stephenfin do you think that would be good to do as a followup? looking at the migrate data object we dont pass any of that info today
16:14:07 gibi sean-k-mooney: do you say that during live migration the server will use the source host config regarding the number of queues?
16:15:16 sean-k-mooney yes and rx/tx queue lenghts
16:15:35 sean-k-mooney we have no support for updating this at all
16:16:16 sean-k-mooney gibi: cold migrate will be fine and a hard reboot would fix it after a live migrate
16:16:52 sean-k-mooney it would only be an issue if you went form a new host with a higher limit to an old host with a smaller one
16:17:27 sean-k-mooney the bug in this case was we limit the queues to 8 eventhogh the kernel support 256 so in generally it wont be an issue
16:17:50 sean-k-mooney but for correct ness we should be using the destination value
16:18:12 gibi sean-k-mooney: yeah, I think that the fix to use the dest value make sense
16:18:30 sean-k-mooney it should be a seperate patch however right
16:18:38 sean-k-mooney so we can backport this one
16:18:57 sean-k-mooney the fix for dest value would need an object change
16:20:00 sean-k-mooney assuming livirt allows it at all. it might just say no an reject the live migration
16:21:09 sean-k-mooney gibi: ill file a bug and we can figure out what to do with it later
16:21:59 gibi sean-k-mooney: ack
16:27:35 efried stephenfin: can I get a quick re+A on https://review.opendev.org/#/c/696354/ ? (Updated UT)
16:32:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656
16:32:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Flesh out docs for cross-cell resize/cold migrate https://review.opendev.org/696212
16:32:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Simplify FinishResizeAtDestTask event handling https://review.opendev.org/695337
16:32:16 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add negative test to delete server during cross-cell resize claim https://review.opendev.org/688832
16:32:16 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Implement reschedule logic for cross-cell resize/migrate https://review.opendev.org/696213
16:32:59 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522
16:33:03 stephenfin Sure, done
16:33:26 stephenfin efried: Some real nice docs there ^ :)
16:35:28 efried thanks, and noted.
16:36:06 dansmith efried: did you see my comment about the event getting 422?
16:36:14 efried not yet
16:36:42 dansmith okay I don't want sundar to be held up on a decision there or anything
16:37:04 efried reading now
16:38:41 efried dansmith: oh, that's a good point. If cyborg gets this particular 422 it should mean the bind finished before we even got to the host, so the poll inside wait_for_instance_event will trigger and prevent us from waiting (for an event that won't come)
16:39:16 dansmith efried: right, if it gets a 404 then it should fail the bind, but if it gets 422, it shouldn't freak out
16:39:25 efried ++
16:39:37 dansmith in fact, 422 confirms, in a weird way, that the instance is still there, just not on a host yet
16:40:16 dansmith so if you could chuck that ++ into the review so sundar isn't waiting that would be good
16:43:30 efried dansmith: done
16:43:52 dansmith efried: spanks
16:44:05 efried comfortable and slimming
16:44:07 dansmith efried: have you chatted with him to know that he's responning?
16:44:11 dansmith *respinning
16:44:18 efried haltingly
16:44:30 dansmith haltingly?
16:44:41 efried think voicemail tag
16:44:45 efried but with slack
16:46:07 dansmith I, uh, will trust you
16:49:53 stephenfin dansmith: Can I remove remotable methods on objects if there are no callers left in-tree
16:50:11 stephenfin Or are they versioned since they're RPC'y
16:50:26 dansmith stephenfin: not according to the letter of the law

Earlier   Later