Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-13
18:20:03 efried melwitt: are you now owning nova-audit? (https://review.opendev.org/#/c/693226/)
18:20:46 melwitt efried: I didn't want to but I think the answer is technically yes because dansmith lost interest
18:21:28 efried melwitt: well, I ask because we're at spec freeze, so you need to get a couple cores on board, ahem, today if it's going to happen in ussuri.
18:22:26 bauzas efried: melwitt: FWIW, this is related https://review.opendev.org/#/c/670112/
18:22:43 efried it is?
18:22:54 bauzas technically, it's just a rename
18:23:08 bauzas but the intent of the spec is to provide a new specific command AFAICR
18:23:20 bauzas this change ^ would just be another subcommand
18:24:34 melwitt efried: yeah, I don't think that's going to happen. operators are interested but the spec didn't attract review from cores thus far and I don't think I could wrangle two that would not be considered part owners by the end of today
18:25:30 efried melwitt: if "tomorrow" would make the difference, I'm fine with that. Or do you just want me to defer?
18:26:28 melwitt bauzas: the intent of the spec is to organize all of the heal commands in one place and make them runnable as a daemon service so that they automatically heal your cloud periodically
18:27:27 bauzas oh missed the last part
18:27:31 bauzas gtk
18:28:01 openstackgerrit Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.opendev.org/552924
18:28:10 bauzas efried: ^
18:28:12 efried ack
18:29:18 bauzas anyway, bailing out
18:29:21 melwitt efried: I guess yeah if you'll give it till tomorrow, I'll send some email and see if anyone's willing to review. if there's not interest after that, then punt it
18:29:34 Sundar dansmith: If https://review.opendev.org/#/c/673735/37/nova/conductor/manager.py@524 is not the right place to delete ARQs on a reschedule, do you have any suggestion for a better plac? I could do it in the callers.
18:29:51 efried melwitt: ack. I'm adding it (with other open specs) to today's meeting agenda, if you want to drum up interest there.
18:29:55 dansmith melwitt: efried it seems highly unlikely that anything would get implemented in U either way, so I'm not sure it's worth that
18:30:16 dansmith I thought we were supposed to be trying to reduce the number of things we approved that aren't likely to make it,
18:30:36 dansmith but it kinda seems like we're doing the same ol' kind of behavior
18:31:18 dansmith Sundar: do it where it needs to be done, not inside a thing called something else.. so yes, wherever that's called from that is the right place
18:32:37 melwitt dansmith, efried: well, I could implement it quickly/dumbly (I'm imagining just moving the commands and adding a service) but getting review would be another story. worst case it sits there ready to go for V if ppl can't review in time. so, I dunno
18:33:21 efried dansmith: Yes, intend to do a sweep of Definition:Approved blueprints "soon" to decide which of those we can/should defer.
18:34:06 efried "spec freeze" -- no more definition approvals -- is what's happening now.
18:37:54 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201
18:45:32 efried johnthetubaguy: Save me looking, did you squash the fup?
18:45:38 efried (abandon if so)
18:51:31 dking_desktop I'm attempting to troubleshoot why I get the "No valid host was found." error when attempting to create a baremetal server, and just found this when I enabled debugging for the nova-scheduler: compute_status_filter request filter added forbidden trait COMPUTE_STATUS_DISABLED
18:52:07 dking_desktop Could that be the reason why I'm not able to find a valid host? How would I troubleshoot this further?
19:03:49 efried dking_desktop: We always add that trait. It's only going to have an effect if the compute host is exposing that trait. You can check with a command like
19:03:49 efried openstack resource provider trait list $host_uuid
19:04:01 efried (I may not have the syntax exactly right -- see the docs)
19:07:00 dking_desktop I'm using Ironic if that helps. I don't see anything for "openstack resource". Is it "openstack service provider list"?
19:08:11 dking_desktop Oh, maybe "openstack baremetal node trait list"
19:09:45 dking_desktop efried: I tried "openstack baremetal node trait list <UUID>", but that gave no results. Is that the problem?
19:10:14 efried dking_desktop: You need to install the osc-placement plugin to get the 'resource provider' subcommands
19:10:28 efried pip install osc-placement (or equivalent for your distro)
19:10:43 efried COMPUTE_STATUS_DISABLED isn't a trait that ironic itself would know about.
19:12:59 dking_desktop Odd. I get "Operation or argument is not supported with version 1.0; requires at least version 1.6"
19:14:45 dking_desktop I wonder what software that refers to. The osc-placement package should be 1.8.0.
19:16:18 dking_desktop python-openstackclient is 4.0.0. Is there another way to check? It's good to know that isn't specifically an Ironic thing. However, my regular VMs work fine. It's only the baremetal nodes causing me trouble.
19:19:47 dking_desktop Oh, I add that to the command line. Okay, I can run that, but no mention of the above trait.
19:21:38 dking_desktop http://paste.openstack.org/show/789544/
19:25:13 dking_desktop efried: I notice that the above output doesn't show nearly as much information as I see for my compute node. Would the problem be that there's just not any information there about the CPU, etc.?
19:26:41 efried dking_desktop: sorry, yes, you need to specify a microversion for almost every OSC command with placement, as OSC defaults to 1.0 and very little in placement worked at that microversion. You can use an environment variable if you'd rather not have to think about it with every command.
19:27:34 efried dking_desktop: When you say "as I see for my compute node", you mean a libvirt host?
19:27:58 efried is that what compute1.stack1 is?
19:30:45 dking_desktop Correct
19:30:53 efried Next thing to look at is the inventory of your ironic node vs. the flavor you're trying to deploy.
19:30:53 efried If you're seeing COMPUTE_STATUS_DISABLED in play, your control plane is at least at Train, which means your node is supposed to be at least at Stein, by which time we had cut ironic over to reporting single-unit custom resource classes.
19:31:28 efried something like
19:31:28 efried openstack resource provider inventory list (or maybe show) $node_uuid
19:31:28 efried should show you that.
19:31:38 dking_desktop I'm using train.
19:32:45 efried okay, so you might want to make life easier with
19:32:45 efried export OS_PLACEMENT_API_VERSION=1.36
19:32:45 efried (I think I'm spelling that var name right)
19:32:45 efried Then you won't have to add --os-placement-api-version with every command.
19:32:50 dking_desktop Okay, that shows me the resource class and a few other pieces of info: http://paste.openstack.org/show/789545/
19:34:06 efried Great. So the flavor you're using should be asking for resources:CUSTOM_BAREMETAL_RESOURCE_CLASS=1
19:34:08 efried is it?
19:35:05 dking_desktop Yes, it is: http://paste.openstack.org/show/789546/
19:35:18 efried the fact that your trait list didn't show COMPUTE_STATUS_DISABLED, and your inventory showed reserved=0, is two out of the three things that should make this node eligible for scheduling.
19:36:17 dking_desktop That sounds good. Any idea why I would be getting "No valid host was found."
19:36:28 efried okay, I need to go check whether you need to do something special the ram and disk (set them to zero) but I think those should be ignored. Meanwhile, the last of the three ^ things is to make sure there's no allocation present.
19:36:45 efried openstack resource provider allocation list (or show?) $node_uuid
19:37:34 dking_desktop That's empty.
19:37:53 efried Okay. Easier than me finding that code would be checking your placement logs.
19:38:25 efried Look for a line that has a GET call to the /allocation_candidates route with a querystring that includes CUSTOM_BAREMETAL_RESOURCE_CLASS
19:40:27 efried Okay, yeah, it looks like you need to set the flavor VCPUs to zero to make this work.
19:41:32 efried dking_desktop: like this: https://docs.openstack.org/ironic/latest/install/configure-nova-flavors
19:42:27 efried (not just VCPU, MEMORY_MB and DISK_GB too)
19:42:33 dking_desktop efried: Sorry, took me a minute to find it: http://paste.openstack.org/show/789547/
19:42:52 dking_desktop Ah, so the VCPUs could be the problem? Let me see if I can update that.
19:43:58 efried Yup, so you see where that query is *also* asking for DISK_GB%3A1%2CMEMORY_MB%3A512%2CVCPU%3A1 (DISK_GB:1,MEMORY_MB:512,VCPU:1)?
19:44:13 efried your baremetal node's inventory doesn't have any of those resources.
19:44:46 efried Those are being fed in from your base flavor's disk/ram/vcpus
19:45:07 efried So the fix (ahem, it's a hack, I am ashamed) is to explicitly override those with zeros to take them out of the query.
19:46:02 efried And I think we did that hack because letting you set the base flavor values to real zeros would have blown up the code in a billion places
19:46:45 efried dking_desktop: anyway, if you follow https://docs.openstack.org/ironic/latest/install/configure-nova-flavors you should be able to make it work.
19:47:38 dking_desktop I'm having trouble getting it to accept 0.
19:48:25 efried dking_desktop: You can't set the base flavor properties to zero. You have to set *additional* extra specs for resources:{VCPU, MEMORY_MB, DISK_GB}
19:48:28 efried ... to zero
19:49:02 efried That will cause the scheduler to ignore the base vcpus/ram/disk values.
19:49:08 efried ...which can be set to whatever.
19:50:41 dking_desktop Ah. Yes, that's what's in the article, so that makes sense. Let me try that.
19:55:49 dking_desktop Great! That got much further. The build still failed, and I need to investigate that, but at least it started trying.
19:56:30 efried Okay, good deal.
19:59:16 dking_desktop I got: 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance cc78edb5-0268-4d71-a48d-61608b532d6f.'
19:59:35 dking_desktop Do you know where in the logs I might see where that failed?
20:01:37 dking_desktop Oh, I see it in nova-compute-ironic.log. It seems that the deploy image needs to be a UUID and not a name.
20:01:58 efried dking_desktop: I assume that was in your controller logs. You want to look in the compute log on the host that owns your ironic node to see why it bounced.
20:02:48 dking_desktop Yeah. nova-compute-ironic.log showed: Validation of image href deploy-initrd failed, reason: Scheme-less image href is not a UUID.
20:06:48 dking_desktop efried: Thank you very much for your help! I wouldn't have been able to make progress today without it.
20:07:11 efried dking_desktop: You're welcome.
20:08:02 efried dking_desktop: Reading between the lines, you're trying $old_release configurations/images/etc against $new_release code, and running into stuff we changed along the way.
20:09:37 efried dking_desktop: since it's ironic you're trying to deploy, you might also try the #openstack-ironic channel, as they'll generally be more familiar with the quirks in that area. dtantsur|afk I think would be particularly helpful, though he's Eastern Europe so best to hit him earlier in the day.

Earlier   Later