Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-25
14:35:02 dansmith yeah
14:40:23 dansmith mriedem: https://pastebin.com/cCYBiD3h
14:40:45 dansmith emitted from conductor after each host
14:42:25 openstackgerrit Eric Fried proposed openstack/os-traits master: Stop testing with py2 and switch to ussuri jobs https://review.opendev.org/690997
14:45:32 openstackgerrit Eric Fried proposed openstack/os-traits master: Stop testing with py2 and switch to ussuri jobs https://review.opendev.org/690997
14:46:52 openstackgerrit Eric Fried proposed openstack/os-resource-classes master: Stop testing with py2 https://review.opendev.org/690996
14:47:42 gregwork has anyone seen a situation where nova is not getting the update from cloudbase-init on the admin password on provision? eg. running nova get-password instance mykey.pem returns nothing occasionally
14:48:00 gregwork in the console log of the instance I see cloudbase set the admin password and notify metadata
14:49:09 mriedem dansmith: seems reasonable
14:55:23 mriedem gregwork: asking in #openstack-hyper-v might be more productive
14:55:35 openstackgerrit Eric Fried proposed openstack/os-vif master: Drop python2 support and testing https://review.opendev.org/691364
14:55:42 mriedem or poking through open/merged changes https://review.opendev.org/#/q/project:x/cloudbase-init
14:55:50 gregwork mriedem: i didnt even know that existed, thanks :)
14:59:13 dansmith gibi: you around?
14:59:24 gibi dansmith: yep
14:59:44 gibi dansmith: I saw your convo with Matt about the instance cache notification
14:59:47 dansmith gibi: I dunno if you noticed any of that between mriedem and I, but I'm moving on with more rich notifications for progress
15:00:12 gibi dansmith: I skimmed the convo (busy with fixing up patches) overall seems good to me
15:00:12 gregwork mriedem: hmmn that channel has 3 people in it including me :/
15:00:46 dansmith gibi: cool, so.. I guess I need a new action for the per-host notification, but it strikes me as a good candidate to instead have a new phase, like "progress" so... start..progress...progress...progress...end
15:00:52 dansmith gibi: I assume we don't have that for some reason
15:01:37 gibi dansmith: there is instance.update for example so I have nothing against adding progress as well
15:02:12 dansmith gibi: but that's an action, not a phase yeah?
15:03:06 gibi dansmith: hm. good point. your notification would be like image_cache.start, image_cache.progress ... image_cache.end
15:03:19 dansmith gibi: right, that's what would make the most sense to me
15:03:23 gibi dansmith: I have no problem with this either
15:03:55 mriedem gregwork: lots of the cloudbase people are in eastern europe i think
15:04:00 mriedem lpetrut is maybe your best bet
15:04:18 dansmith gibi: okay I'll add it and you can see what you think in the review
15:04:28 gregwork so this is definitely a clousbase agent problem and not a wierd nova issue i ran into ?
15:04:30 gibi dansmith: sure. I will check
15:04:37 gregwork *cloudbase
15:04:37 dansmith gibi: thanks
15:04:49 mriedem gregwork: i have no idea
15:05:06 mriedem but i'd start there rather than here
15:05:29 gregwork ok thanks for the contact info
15:19:44 openstackgerrit Eric Fried proposed openstack/python-novaclient master: Stop supporting and testing python2 https://review.opendev.org/691365
15:23:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387
15:24:45 gibi mriedem: one up ^^ one more to go
15:32:20 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
15:32:48 gibi mriedem: done as well ^^
15:34:00 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
15:34:03 gibi this marks the end of my work week. have a nice weekend folks!
15:37:03 dansmith gibi: wait
15:37:05 dansmith WAIT
15:37:45 dansmith gibi: in functional, I'm doing wait_for_versioned_notification() by the things it returns have empty payloads.. do I need to enable some config to have that work?
15:40:09 dansmith mriedem: do you know?
15:41:25 mriedem no, you just need to stub the fake notifier stuff
15:41:46 dansmith okay, I got that of course and am getting the notifs but payloads are empty
15:41:50 dansmith maybe I'm emitting wrong
15:46:22 dansmith must be because the other ones have payloads
15:46:23 dansmith danigt
15:48:07 dansmith helps if I register the payload too :D
15:48:47 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Fail compute service delete if resource provider delete fails https://review.opendev.org/678100
15:49:58 mriedem ^ is wip until i get some reviews on what's proposed there which is based on what was discussed in the ML for not orphaning resource providers when deleting a compute service that still has allocations against it
15:50:15 mriedem we continue to get bugs in triage for this kind of thign
15:50:28 mriedem e.g. https://bugs.launchpad.net/nova/+bug/1849701
15:50:29 openstack Launchpad bug 1849701 in OpenStack Compute (nova) "Resource_provider entry related to a deleted compute node, unable to migrate vms to the node" [Undecided,New]
16:06:45 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
16:09:33 mriedem looks like docs job is busted
16:10:23 cdent dansmith, mriedem : when nova-compute comes up and tries to initiate talking to the conductor it uses an "early_timeout" and then after 10 attempts fails over to the global timeout. If those 10 attempts pass, is it possible the nova-compute will sit doing nothing on that long timeout for a long time before talking to the conductor, or should it loop quickly if there is an actual connection failed (rather than timing out
16:10:23 cdent waiting for response)? I'm reading some bugs where it seems like the sit doing nothing is happening, but it seems illogical.
16:10:24 mriedem https://zuul.opendev.org/t/openstack/build/10344924bca44849b06b8fae29b5494f/log/job-output.txt#652
16:11:22 mriedem cdent: not sure off the top of my head, probably a better question for dansmith
16:13:25 cdent in placement we addressed that docs issue by having a different set of requirements for docs
16:13:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
16:14:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387
16:14:08 dansmith cdent: not sure what you mean by early timeout
16:14:18 cdent dansmith: one sec, will get code link
16:14:20 dansmith cdent: it should be calling ping for the conductor pretty regularly and wait forever
16:14:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688
16:15:30 cdent dansmith: https://github.com/openstack/nova/blob/master/nova/conductor/api.py#L41
16:16:05 dansmith oh right
16:16:11 cdent after early attempts expires, if the global rpc timeout is huge (because people do that...) it appears like it can be a very long time before "ready" happens
16:16:43 cdent but I would have thought that if the conductor is literally not there, you would know sooner
16:16:50 dansmith cdent: people who set the global timeout large are begging for problems, so I don't think we should do something different here
16:16:51 dansmith however,
16:17:25 mriedem vmware sets the timeout large because they are pulling 1K+ instances from conductor on startup yeah?
16:17:26 cdent but could it be that if the message bus is up but the conductor is not, the ping will hang out in rabbit... waiting
16:17:28 dansmith the reason for that was because if you have a power outage and start a thousand computes, they need to chill if they can't get a response right away.. just because they have a connection doesn't mean they're not overwhelming the conductor(s), which is why we have the backoff
16:17:44 dansmith cdent: sure
16:17:47 cdent mriedem: probably something like that, yeah
16:18:13 cdent Okay, cool, this confirms what I thought was happening, but hadn't been able to trace it all the way down
16:18:36 cdent mriedem: there also appears to be an ordering problem in service restarts
16:19:01 cdent thanks to you both
16:19:10 cdent I'll go write down more stuff that people will ignore...
16:19:17 cdent (not you, I mean internal people)
16:19:32 mriedem *those* people
16:19:35 openstackgerrit John Garbutt proposed openstack/nova master: WIP: Integrating with unified limits https://review.opendev.org/615180
16:19:51 cdent them
16:24:10 mriedem blarg, yeah we have doc/requirements.txt but https://review.opendev.org/#/c/689152/ threw requirements.txt back into the install for the docs job
16:24:11 mriedem fixing
16:26:36 openstackgerrit Dan Smith proposed openstack/nova master: Log some stats for image pre-cache https://review.opendev.org/688173
16:26:37 openstackgerrit Dan Smith proposed openstack/nova master: Add Aggregate image caching progress notifications https://review.opendev.org/691390
16:29:03 mriedem https://bugs.launchpad.net/nova/+bug/1849870
16:29:03 openstack Launchpad bug 1849870 in OpenStack Compute (nova) "docs build failing on psycopg2 dependency" [Undecided,New]
16:39:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Stop building docs with (test-)requirements.txt https://review.opendev.org/691391
16:39:43 mriedem ^ will hopefully unblock the gate for nova ^
16:41:48 dansmith mriedem: should we slam that in?
16:42:11 mriedem let's see if it passes the docs job first, it works for me locally

Earlier   Later