Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-25
14:31:20 dansmith either way,
14:31:35 dansmith 2 with a total and complete count makes the most sense to me I think
14:31:50 mriedem having the aggregate in the per-compute notification is fine i think
14:31:53 dansmith I wasn't sure if once per compute was too much, but sounds like you're thinking that way
14:32:06 mriedem you'd have the aggregate, the compute, maybe the image(s) and the status
14:32:16 mriedem we could make it configurable
14:32:22 mriedem like we do for bdms in notifications
14:32:32 mriedem so by default we don't send all that noise, but if you need it it's there
14:32:53 mriedem or we wait until a consumer comes to us with their actual requirement for what they need to get out of thise
14:32:54 mriedem *this
14:33:08 dansmith you really kinda need to know what the status of the images are too if you want this to be coherent
14:33:22 dansmith well, I have one customer requirement so I have some of that
14:33:49 dansmith lemme run with 2+info and we can argue about the contents and/or what should be included by default or by config in gerrit
14:34:18 mriedem fetching the image status....that means doing yet another GET on each image after we've validated them in the API which would make me sad
14:34:28 dansmith no,
14:34:35 dansmith I mean the result of the operation from thecompute
14:34:51 mriedem oh yeah that's fine, i said that'd be in there above
14:35:02 dansmith yeah
14:40:23 dansmith mriedem: https://pastebin.com/cCYBiD3h
14:40:45 dansmith emitted from conductor after each host
14:42:25 openstackgerrit Eric Fried proposed openstack/os-traits master: Stop testing with py2 and switch to ussuri jobs https://review.opendev.org/690997
14:45:32 openstackgerrit Eric Fried proposed openstack/os-traits master: Stop testing with py2 and switch to ussuri jobs https://review.opendev.org/690997
14:46:52 openstackgerrit Eric Fried proposed openstack/os-resource-classes master: Stop testing with py2 https://review.opendev.org/690996
14:47:42 gregwork has anyone seen a situation where nova is not getting the update from cloudbase-init on the admin password on provision? eg. running nova get-password instance mykey.pem returns nothing occasionally
14:48:00 gregwork in the console log of the instance I see cloudbase set the admin password and notify metadata
14:49:09 mriedem dansmith: seems reasonable
14:55:23 mriedem gregwork: asking in #openstack-hyper-v might be more productive
14:55:35 openstackgerrit Eric Fried proposed openstack/os-vif master: Drop python2 support and testing https://review.opendev.org/691364
14:55:42 mriedem or poking through open/merged changes https://review.opendev.org/#/q/project:x/cloudbase-init
14:55:50 gregwork mriedem: i didnt even know that existed, thanks :)
14:59:13 dansmith gibi: you around?
14:59:24 gibi dansmith: yep
14:59:44 gibi dansmith: I saw your convo with Matt about the instance cache notification
14:59:47 dansmith gibi: I dunno if you noticed any of that between mriedem and I, but I'm moving on with more rich notifications for progress
15:00:12 gibi dansmith: I skimmed the convo (busy with fixing up patches) overall seems good to me
15:00:12 gregwork mriedem: hmmn that channel has 3 people in it including me :/
15:00:46 dansmith gibi: cool, so.. I guess I need a new action for the per-host notification, but it strikes me as a good candidate to instead have a new phase, like "progress" so... start..progress...progress...progress...end
15:00:52 dansmith gibi: I assume we don't have that for some reason
15:01:37 gibi dansmith: there is instance.update for example so I have nothing against adding progress as well
15:02:12 dansmith gibi: but that's an action, not a phase yeah?
15:03:06 gibi dansmith: hm. good point. your notification would be like image_cache.start, image_cache.progress ... image_cache.end
15:03:19 dansmith gibi: right, that's what would make the most sense to me
15:03:23 gibi dansmith: I have no problem with this either
15:03:55 mriedem gregwork: lots of the cloudbase people are in eastern europe i think
15:04:00 mriedem lpetrut is maybe your best bet
15:04:18 dansmith gibi: okay I'll add it and you can see what you think in the review
15:04:28 gregwork so this is definitely a clousbase agent problem and not a wierd nova issue i ran into ?
15:04:30 gibi dansmith: sure. I will check
15:04:37 gregwork *cloudbase
15:04:37 dansmith gibi: thanks
15:04:49 mriedem gregwork: i have no idea
15:05:06 mriedem but i'd start there rather than here
15:05:29 gregwork ok thanks for the contact info
15:19:44 openstackgerrit Eric Fried proposed openstack/python-novaclient master: Stop supporting and testing python2 https://review.opendev.org/691365
15:23:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387
15:24:45 gibi mriedem: one up ^^ one more to go
15:32:20 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
15:32:48 gibi mriedem: done as well ^^
15:34:00 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
15:34:03 gibi this marks the end of my work week. have a nice weekend folks!
15:37:03 dansmith gibi: wait
15:37:05 dansmith WAIT
15:37:45 dansmith gibi: in functional, I'm doing wait_for_versioned_notification() by the things it returns have empty payloads.. do I need to enable some config to have that work?
15:40:09 dansmith mriedem: do you know?
15:41:25 mriedem no, you just need to stub the fake notifier stuff
15:41:46 dansmith okay, I got that of course and am getting the notifs but payloads are empty
15:41:50 dansmith maybe I'm emitting wrong
15:46:22 dansmith must be because the other ones have payloads
15:46:23 dansmith danigt
15:48:07 dansmith helps if I register the payload too :D
15:48:47 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Fail compute service delete if resource provider delete fails https://review.opendev.org/678100
15:49:58 mriedem ^ is wip until i get some reviews on what's proposed there which is based on what was discussed in the ML for not orphaning resource providers when deleting a compute service that still has allocations against it
15:50:15 mriedem we continue to get bugs in triage for this kind of thign
15:50:28 mriedem e.g. https://bugs.launchpad.net/nova/+bug/1849701
15:50:29 openstack Launchpad bug 1849701 in OpenStack Compute (nova) "Resource_provider entry related to a deleted compute node, unable to migrate vms to the node" [Undecided,New]
16:06:45 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
16:09:33 mriedem looks like docs job is busted
16:10:23 cdent dansmith, mriedem : when nova-compute comes up and tries to initiate talking to the conductor it uses an "early_timeout" and then after 10 attempts fails over to the global timeout. If those 10 attempts pass, is it possible the nova-compute will sit doing nothing on that long timeout for a long time before talking to the conductor, or should it loop quickly if there is an actual connection failed (rather than timing out
16:10:23 cdent waiting for response)? I'm reading some bugs where it seems like the sit doing nothing is happening, but it seems illogical.
16:10:24 mriedem https://zuul.opendev.org/t/openstack/build/10344924bca44849b06b8fae29b5494f/log/job-output.txt#652
16:11:22 mriedem cdent: not sure off the top of my head, probably a better question for dansmith
16:13:25 cdent in placement we addressed that docs issue by having a different set of requirements for docs
16:13:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
16:14:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387
16:14:08 dansmith cdent: not sure what you mean by early timeout
16:14:18 cdent dansmith: one sec, will get code link
16:14:20 dansmith cdent: it should be calling ping for the conductor pretty regularly and wait forever
16:14:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
16:15:30 cdent dansmith: https://github.com/openstack/nova/blob/master/nova/conductor/api.py#L41
16:16:05 dansmith oh right
16:16:11 cdent after early attempts expires, if the global rpc timeout is huge (because people do that...) it appears like it can be a very long time before "ready" happens
16:16:43 cdent but I would have thought that if the conductor is literally not there, you would know sooner
16:16:50 dansmith cdent: people who set the global timeout large are begging for problems, so I don't think we should do something different here
16:16:51 dansmith however,
16:17:25 mriedem vmware sets the timeout large because they are pulling 1K+ instances from conductor on startup yeah?
16:17:26 cdent but could it be that if the message bus is up but the conductor is not, the ping will hang out in rabbit... waiting
16:17:28 dansmith the reason for that was because if you have a power outage and start a thousand computes, they need to chill if they can't get a response right away.. just because they have a connection doesn't mean they're not overwhelming the conductor(s), which is why we have the backoff
16:17:44 dansmith cdent: sure
16:17:47 cdent mriedem: probably something like that, yeah
16:18:13 cdent Okay, cool, this confirms what I thought was happening, but hadn't been able to trace it all the way down

Earlier   Later