| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 18:34:06 | efried | "spec freeze" -- no more definition approvals -- is what's happening now. | |
| 18:37:54 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201 | |
| 18:45:32 | efried | johnthetubaguy: Save me looking, did you squash the fup? | |
| 18:45:38 | efried | (abandon if so) | |
| 18:51:31 | dking_desktop | I'm attempting to troubleshoot why I get the "No valid host was found." error when attempting to create a baremetal server, and just found this when I enabled debugging for the nova-scheduler: compute_status_filter request filter added forbidden trait COMPUTE_STATUS_DISABLED | |
| 18:52:07 | dking_desktop | Could that be the reason why I'm not able to find a valid host? How would I troubleshoot this further? | |
| 19:03:49 | efried | dking_desktop: We always add that trait. It's only going to have an effect if the compute host is exposing that trait. You can check with a command like | |
| 19:03:49 | efried | openstack resource provider trait list $host_uuid | |
| 19:04:01 | efried | (I may not have the syntax exactly right -- see the docs) | |
| 19:07:00 | dking_desktop | I'm using Ironic if that helps. I don't see anything for "openstack resource". Is it "openstack service provider list"? | |
| 19:08:11 | dking_desktop | Oh, maybe "openstack baremetal node trait list" | |
| 19:09:45 | dking_desktop | efried: I tried "openstack baremetal node trait list <UUID>", but that gave no results. Is that the problem? | |
| 19:10:14 | efried | dking_desktop: You need to install the osc-placement plugin to get the 'resource provider' subcommands | |
| 19:10:28 | efried | pip install osc-placement (or equivalent for your distro) | |
| 19:10:43 | efried | COMPUTE_STATUS_DISABLED isn't a trait that ironic itself would know about. | |
| 19:12:59 | dking_desktop | Odd. I get "Operation or argument is not supported with version 1.0; requires at least version 1.6" | |
| 19:14:45 | dking_desktop | I wonder what software that refers to. The osc-placement package should be 1.8.0. | |
| 19:16:18 | dking_desktop | python-openstackclient is 4.0.0. Is there another way to check? It's good to know that isn't specifically an Ironic thing. However, my regular VMs work fine. It's only the baremetal nodes causing me trouble. | |
| 19:19:47 | dking_desktop | Oh, I add that to the command line. Okay, I can run that, but no mention of the above trait. | |
| 19:21:38 | dking_desktop | http://paste.openstack.org/show/789544/ | |
| 19:25:13 | dking_desktop | efried: I notice that the above output doesn't show nearly as much information as I see for my compute node. Would the problem be that there's just not any information there about the CPU, etc.? | |
| 19:26:41 | efried | dking_desktop: sorry, yes, you need to specify a microversion for almost every OSC command with placement, as OSC defaults to 1.0 and very little in placement worked at that microversion. You can use an environment variable if you'd rather not have to think about it with every command. | |
| 19:27:34 | efried | dking_desktop: When you say "as I see for my compute node", you mean a libvirt host? | |
| 19:27:58 | efried | is that what compute1.stack1 is? | |
| 19:30:45 | dking_desktop | Correct | |
| 19:30:53 | efried | Next thing to look at is the inventory of your ironic node vs. the flavor you're trying to deploy. | |
| 19:30:53 | efried | If you're seeing COMPUTE_STATUS_DISABLED in play, your control plane is at least at Train, which means your node is supposed to be at least at Stein, by which time we had cut ironic over to reporting single-unit custom resource classes. | |
| 19:31:28 | efried | something like | |
| 19:31:28 | efried | openstack resource provider inventory list (or maybe show) $node_uuid | |
| 19:31:28 | efried | should show you that. | |
| 19:31:38 | dking_desktop | I'm using train. | |
| 19:32:45 | efried | okay, so you might want to make life easier with | |
| 19:32:45 | efried | export OS_PLACEMENT_API_VERSION=1.36 | |
| 19:32:45 | efried | (I think I'm spelling that var name right) | |
| 19:32:45 | efried | Then you won't have to add --os-placement-api-version with every command. | |
| 19:32:50 | dking_desktop | Okay, that shows me the resource class and a few other pieces of info: http://paste.openstack.org/show/789545/ | |
| 19:34:06 | efried | Great. So the flavor you're using should be asking for resources:CUSTOM_BAREMETAL_RESOURCE_CLASS=1 | |
| 19:34:08 | efried | is it? | |
| 19:35:05 | dking_desktop | Yes, it is: http://paste.openstack.org/show/789546/ | |
| 19:35:18 | efried | the fact that your trait list didn't show COMPUTE_STATUS_DISABLED, and your inventory showed reserved=0, is two out of the three things that should make this node eligible for scheduling. | |
| 19:36:17 | dking_desktop | That sounds good. Any idea why I would be getting "No valid host was found." | |
| 19:36:28 | efried | okay, I need to go check whether you need to do something special the ram and disk (set them to zero) but I think those should be ignored. Meanwhile, the last of the three ^ things is to make sure there's no allocation present. | |
| 19:36:45 | efried | openstack resource provider allocation list (or show?) $node_uuid | |
| 19:37:34 | dking_desktop | That's empty. | |
| 19:37:53 | efried | Okay. Easier than me finding that code would be checking your placement logs. | |
| 19:38:25 | efried | Look for a line that has a GET call to the /allocation_candidates route with a querystring that includes CUSTOM_BAREMETAL_RESOURCE_CLASS | |
| 19:40:27 | efried | Okay, yeah, it looks like you need to set the flavor VCPUs to zero to make this work. | |
| 19:41:32 | efried | dking_desktop: like this: https://docs.openstack.org/ironic/latest/install/configure-nova-flavors | |
| 19:42:27 | efried | (not just VCPU, MEMORY_MB and DISK_GB too) | |
| 19:42:33 | dking_desktop | efried: Sorry, took me a minute to find it: http://paste.openstack.org/show/789547/ | |
| 19:42:52 | dking_desktop | Ah, so the VCPUs could be the problem? Let me see if I can update that. | |
| 19:43:58 | efried | Yup, so you see where that query is *also* asking for DISK_GB%3A1%2CMEMORY_MB%3A512%2CVCPU%3A1 (DISK_GB:1,MEMORY_MB:512,VCPU:1)? | |
| 19:44:13 | efried | your baremetal node's inventory doesn't have any of those resources. | |
| 19:44:46 | efried | Those are being fed in from your base flavor's disk/ram/vcpus | |
| 19:45:07 | efried | So the fix (ahem, it's a hack, I am ashamed) is to explicitly override those with zeros to take them out of the query. | |
| 19:46:02 | efried | And I think we did that hack because letting you set the base flavor values to real zeros would have blown up the code in a billion places | |
| 19:46:45 | efried | dking_desktop: anyway, if you follow https://docs.openstack.org/ironic/latest/install/configure-nova-flavors you should be able to make it work. | |
| 19:47:38 | dking_desktop | I'm having trouble getting it to accept 0. | |
| 19:48:25 | efried | dking_desktop: You can't set the base flavor properties to zero. You have to set *additional* extra specs for resources:{VCPU, MEMORY_MB, DISK_GB} | |
| 19:48:28 | efried | ... to zero | |
| 19:49:02 | efried | That will cause the scheduler to ignore the base vcpus/ram/disk values. | |
| 19:49:08 | efried | ...which can be set to whatever. | |
| 19:50:41 | dking_desktop | Ah. Yes, that's what's in the article, so that makes sense. Let me try that. | |
| 19:55:49 | dking_desktop | Great! That got much further. The build still failed, and I need to investigate that, but at least it started trying. | |
| 19:56:30 | efried | Okay, good deal. | |
| 19:59:16 | dking_desktop | I got: 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance cc78edb5-0268-4d71-a48d-61608b532d6f.' | |
| 19:59:35 | dking_desktop | Do you know where in the logs I might see where that failed? | |
| 20:01:37 | dking_desktop | Oh, I see it in nova-compute-ironic.log. It seems that the deploy image needs to be a UUID and not a name. | |
| 20:01:58 | efried | dking_desktop: I assume that was in your controller logs. You want to look in the compute log on the host that owns your ironic node to see why it bounced. | |
| 20:02:48 | dking_desktop | Yeah. nova-compute-ironic.log showed: Validation of image href deploy-initrd failed, reason: Scheme-less image href is not a UUID. | |
| 20:06:48 | dking_desktop | efried: Thank you very much for your help! I wouldn't have been able to make progress today without it. | |
| 20:07:11 | efried | dking_desktop: You're welcome. | |
| 20:08:02 | efried | dking_desktop: Reading between the lines, you're trying $old_release configurations/images/etc against $new_release code, and running into stuff we changed along the way. | |
| 20:09:37 | efried | dking_desktop: since it's ironic you're trying to deploy, you might also try the #openstack-ironic channel, as they'll generally be more familiar with the quirks in that area. dtantsur|afk I think would be particularly helpful, though he's Eastern Europe so best to hit him earlier in the day. | |
| 20:10:25 | dking_desktop | Earlier, I think I wasn't able to find any good documentation. My google searches kept taking me to outdated documentation, and so I think I started using that, only to find trouble down the road after I'd forgotten where I'd found the information. | |
| 20:10:39 | efried | ugh | |
| 20:11:08 | dking_desktop | And yes, the timezone issues make things difficult. I do most of my work while they're sleeping, though they're helpful early in my day. | |
| 20:11:26 | efried | well, what might work *sometimes* is, if you hit a page under docs.openstack.org/$proj/$rele/..., try replacing $rele with `latest` | |
| 20:11:59 | efried | You'll either get the latest instructions, or you'll get a 404, which means whatever you're trying to do won't work anyway :P | |
| 20:12:08 | efried | (in your case s/$rele/train/) | |
| 20:12:43 | efried | bauzas, sean-k-mooney, stephenfin: Responded on https://review.opendev.org/#/c/552924/ (NUMA RPs). | |
| 20:13:35 | efried | I would update it myself, but then stephenfin and gibi would have to be the approvers. | |
| 20:13:47 | efried | I guess we pretty much have to wait for morning anyway, so I'll just leave it. | |
| 20:15:13 | dking_desktop | Thank you so much. I think that now that I'm seeing the correct documentation, I may just start over and read each piece again in order. | |
| 20:16:11 | efried | dking_desktop: Cool. If in doing that you find discrepancies (or even typos), please propose edits. | |
| 20:16:37 | efried | ...or open bugs | |
| 20:16:47 | efried | ...or at least shout in here and we'll help you do ^ | |
| 20:17:19 | dking_desktop | Thanks. I submitted some a while back, and I've been the inspiration for a few more fixes already. Even if I'm having trouble, at least, there's updates being made somewhere. | |
| 20:30:21 | efried | Nova meeting in half an hour in #openstack-meeting. | |
| 20:30:21 | efried | Lots to discuss. Please plan to attend. | |
| 20:33:04 | sean-k-mooney | efried: sorry i was picking up the keys to my now house this afternoon. so i missed the spec discussions. im just getting dinner now but ill try and look at them in an hour or so. | |
| 20:33:18 | sean-k-mooney | i can try and attent the nova meeting too | |
| 20:35:29 | efried | sean-k-mooney: congrats on the house | |
| 20:36:02 | sean-k-mooney | now all i need is furniture, broadband, utilities and to move all my stuff :) | |
| 20:41:30 | efried | sean-k-mooney: would be nice to have a nova-side delegate who attended the vol local cache meeting. | |
| 20:41:56 | efried | I'm watching the replay, but I'm afraid it's not going to help me understand whether this is going to fly for U. | |
| 20:42:26 | efried | I assume lyarwood is sleeping? | |
| 20:43:58 | efried | sean-k-mooney: also, I added you to the os-vif release patch. Assume no reason not to merge that? https://review.opendev.org/707018 | |
| 21:10:13 | sean-k-mooney | not that i know of. i will check the review queue and +1 it soon | |
| 21:33:59 | sean-k-mooney | melwitt: it sound like the quota/flavor counting is boraderline a bug fix but ya if you think its better to do in V then thats fine | |