Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-25
14:18:25 dansmith 2. Have the conductor emit one notification per compute after it reports what it did
14:18:59 dansmith 3. Have conductor emit a progress notification as it is doing it with numbers that can be turned into percentage (like 30% complete or N/M complete)
14:19:26 dansmith I can see benefits and drawbacks to all of those approaches
14:19:57 dansmith 1-2 don't let you predict completion, 3 doesn't given you much info, like which ones failed
14:20:55 openstackgerrit Eric Fried proposed openstack/os-traits master: Stop testing with py2 https://review.opendev.org/690997
14:22:06 openstackgerrit Eric Fried proposed openstack/os-resource-classes master: Stop testing with py2 https://review.opendev.org/690996
14:24:52 openstackgerrit Eric Fried proposed openstack/os-vif master: Drop python2 support and testing https://review.opendev.org/691364
14:28:09 mriedem dansmith: couldn't 2 and 3 be combined?
14:28:28 openstackgerrit Eric Fried proposed openstack/python-novaclient master: Stop supporting and testing python2 https://review.opendev.org/691365
14:28:36 dansmith mriedem: ah, so per compute, but with progress? sure
14:28:48 mriedem also, i think you can do 3 without 3 using the existing notification,
14:29:02 mriedem so you'd listen for aggregate.cache_images.start which passes the list of hosts being processed already,
14:29:21 mriedem then you listen for per-compute notifications for that aggregate, and since you know how many are in the aggregate from the first notification, you can do your own progress tracking
14:29:42 mriedem you know you're done when you get aggregate.cache_images.end
14:29:51 mriedem so i think i'd just go with 2
14:30:09 dansmith that is assuming you don't have multiple operations going at once and/or that you don't have overlapping aggregates, both of which I think are not reasonable
14:30:11 mriedem 1 isn't great if the compute is down
14:30:14 dansmith not reasonable assumptions I mean
14:30:39 mriedem the initial notification has the aggregate name in it though
14:30:54 mriedem so even if you have hosts that overlap aggregates i don't think that's a problem from a tracking pov
14:31:19 dansmith you have to have aggregate in each notification to link all that up
14:31:20 dansmith either way,
14:31:35 dansmith 2 with a total and complete count makes the most sense to me I think
14:31:50 mriedem having the aggregate in the per-compute notification is fine i think
14:31:53 dansmith I wasn't sure if once per compute was too much, but sounds like you're thinking that way
14:32:06 mriedem you'd have the aggregate, the compute, maybe the image(s) and the status
14:32:16 mriedem we could make it configurable
14:32:22 mriedem like we do for bdms in notifications
14:32:32 mriedem so by default we don't send all that noise, but if you need it it's there
14:32:53 mriedem or we wait until a consumer comes to us with their actual requirement for what they need to get out of thise
14:32:54 mriedem *this
14:33:08 dansmith you really kinda need to know what the status of the images are too if you want this to be coherent
14:33:22 dansmith well, I have one customer requirement so I have some of that
14:33:49 dansmith lemme run with 2+info and we can argue about the contents and/or what should be included by default or by config in gerrit
14:34:18 mriedem fetching the image status....that means doing yet another GET on each image after we've validated them in the API which would make me sad
14:34:28 dansmith no,
14:34:35 dansmith I mean the result of the operation from thecompute
14:34:51 mriedem oh yeah that's fine, i said that'd be in there above
14:35:02 dansmith yeah
14:40:23 dansmith mriedem: https://pastebin.com/cCYBiD3h
14:40:45 dansmith emitted from conductor after each host
14:42:25 openstackgerrit Eric Fried proposed openstack/os-traits master: Stop testing with py2 and switch to ussuri jobs https://review.opendev.org/690997
14:45:32 openstackgerrit Eric Fried proposed openstack/os-traits master: Stop testing with py2 and switch to ussuri jobs https://review.opendev.org/690997
14:46:52 openstackgerrit Eric Fried proposed openstack/os-resource-classes master: Stop testing with py2 https://review.opendev.org/690996
14:47:42 gregwork has anyone seen a situation where nova is not getting the update from cloudbase-init on the admin password on provision? eg. running nova get-password instance mykey.pem returns nothing occasionally
14:48:00 gregwork in the console log of the instance I see cloudbase set the admin password and notify metadata
14:49:09 mriedem dansmith: seems reasonable
14:55:23 mriedem gregwork: asking in #openstack-hyper-v might be more productive
14:55:35 openstackgerrit Eric Fried proposed openstack/os-vif master: Drop python2 support and testing https://review.opendev.org/691364
14:55:42 mriedem or poking through open/merged changes https://review.opendev.org/#/q/project:x/cloudbase-init
14:55:50 gregwork mriedem: i didnt even know that existed, thanks :)
14:59:13 dansmith gibi: you around?
14:59:24 gibi dansmith: yep
14:59:44 gibi dansmith: I saw your convo with Matt about the instance cache notification
14:59:47 dansmith gibi: I dunno if you noticed any of that between mriedem and I, but I'm moving on with more rich notifications for progress
15:00:12 gibi dansmith: I skimmed the convo (busy with fixing up patches) overall seems good to me
15:00:12 gregwork mriedem: hmmn that channel has 3 people in it including me :/
15:00:46 dansmith gibi: cool, so.. I guess I need a new action for the per-host notification, but it strikes me as a good candidate to instead have a new phase, like "progress" so... start..progress...progress...progress...end
15:00:52 dansmith gibi: I assume we don't have that for some reason
15:01:37 gibi dansmith: there is instance.update for example so I have nothing against adding progress as well
15:02:12 dansmith gibi: but that's an action, not a phase yeah?
15:03:06 gibi dansmith: hm. good point. your notification would be like image_cache.start, image_cache.progress ... image_cache.end
15:03:19 dansmith gibi: right, that's what would make the most sense to me
15:03:23 gibi dansmith: I have no problem with this either
15:03:55 mriedem gregwork: lots of the cloudbase people are in eastern europe i think
15:04:00 mriedem lpetrut is maybe your best bet
15:04:18 dansmith gibi: okay I'll add it and you can see what you think in the review
15:04:28 gregwork so this is definitely a clousbase agent problem and not a wierd nova issue i ran into ?
15:04:30 gibi dansmith: sure. I will check
15:04:37 gregwork *cloudbase
15:04:37 dansmith gibi: thanks
15:04:49 mriedem gregwork: i have no idea
15:05:06 mriedem but i'd start there rather than here
15:05:29 gregwork ok thanks for the contact info
15:19:44 openstackgerrit Eric Fried proposed openstack/python-novaclient master: Stop supporting and testing python2 https://review.opendev.org/691365
15:23:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387
15:24:45 gibi mriedem: one up ^^ one more to go
15:32:20 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
15:32:48 gibi mriedem: done as well ^^
15:34:00 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
15:34:03 gibi this marks the end of my work week. have a nice weekend folks!
15:37:03 dansmith gibi: wait
15:37:05 dansmith WAIT
15:37:45 dansmith gibi: in functional, I'm doing wait_for_versioned_notification() by the things it returns have empty payloads.. do I need to enable some config to have that work?
15:40:09 dansmith mriedem: do you know?
15:41:25 mriedem no, you just need to stub the fake notifier stuff
15:41:46 dansmith okay, I got that of course and am getting the notifs but payloads are empty
15:41:50 dansmith maybe I'm emitting wrong
15:46:22 dansmith must be because the other ones have payloads
15:46:23 dansmith danigt
15:48:07 dansmith helps if I register the payload too :D
15:48:47 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Fail compute service delete if resource provider delete fails https://review.opendev.org/678100
15:49:58 mriedem ^ is wip until i get some reviews on what's proposed there which is based on what was discussed in the ML for not orphaning resource providers when deleting a compute service that still has allocations against it
15:50:15 mriedem we continue to get bugs in triage for this kind of thign
15:50:28 mriedem e.g. https://bugs.launchpad.net/nova/+bug/1849701
15:50:29 openstack Launchpad bug 1849701 in OpenStack Compute (nova) "Resource_provider entry related to a deleted compute node, unable to migrate vms to the node" [Undecided,New]
16:06:45 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
16:09:33 mriedem looks like docs job is busted
16:10:23 cdent dansmith, mriedem : when nova-compute comes up and tries to initiate talking to the conductor it uses an "early_timeout" and then after 10 attempts fails over to the global timeout. If those 10 attempts pass, is it possible the nova-compute will sit doing nothing on that long timeout for a long time before talking to the conductor, or should it loop quickly if there is an actual connection failed (rather than timing out
16:10:23 cdent waiting for response)? I'm reading some bugs where it seems like the sit doing nothing is happening, but it seems illogical.

Earlier   Later