| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 19:16:18 | dking_desktop | python-openstackclient is 4.0.0. Is there another way to check? It's good to know that isn't specifically an Ironic thing. However, my regular VMs work fine. It's only the baremetal nodes causing me trouble. | |
| 19:19:47 | dking_desktop | Oh, I add that to the command line. Okay, I can run that, but no mention of the above trait. | |
| 19:21:38 | dking_desktop | http://paste.openstack.org/show/789544/ | |
| 19:25:13 | dking_desktop | efried: I notice that the above output doesn't show nearly as much information as I see for my compute node. Would the problem be that there's just not any information there about the CPU, etc.? | |
| 19:26:41 | efried | dking_desktop: sorry, yes, you need to specify a microversion for almost every OSC command with placement, as OSC defaults to 1.0 and very little in placement worked at that microversion. You can use an environment variable if you'd rather not have to think about it with every command. | |
| 19:27:34 | efried | dking_desktop: When you say "as I see for my compute node", you mean a libvirt host? | |
| 19:27:58 | efried | is that what compute1.stack1 is? | |
| 19:30:45 | dking_desktop | Correct | |
| 19:30:53 | efried | Next thing to look at is the inventory of your ironic node vs. the flavor you're trying to deploy. | |
| 19:30:53 | efried | If you're seeing COMPUTE_STATUS_DISABLED in play, your control plane is at least at Train, which means your node is supposed to be at least at Stein, by which time we had cut ironic over to reporting single-unit custom resource classes. | |
| 19:31:28 | efried | something like | |
| 19:31:28 | efried | openstack resource provider inventory list (or maybe show) $node_uuid | |
| 19:31:28 | efried | should show you that. | |
| 19:31:38 | dking_desktop | I'm using train. | |
| 19:32:45 | efried | okay, so you might want to make life easier with | |
| 19:32:45 | efried | export OS_PLACEMENT_API_VERSION=1.36 | |
| 19:32:45 | efried | (I think I'm spelling that var name right) | |
| 19:32:45 | efried | Then you won't have to add --os-placement-api-version with every command. | |
| 19:32:50 | dking_desktop | Okay, that shows me the resource class and a few other pieces of info: http://paste.openstack.org/show/789545/ | |
| 19:34:06 | efried | Great. So the flavor you're using should be asking for resources:CUSTOM_BAREMETAL_RESOURCE_CLASS=1 | |
| 19:34:08 | efried | is it? | |
| 19:35:05 | dking_desktop | Yes, it is: http://paste.openstack.org/show/789546/ | |
| 19:35:18 | efried | the fact that your trait list didn't show COMPUTE_STATUS_DISABLED, and your inventory showed reserved=0, is two out of the three things that should make this node eligible for scheduling. | |
| 19:36:17 | dking_desktop | That sounds good. Any idea why I would be getting "No valid host was found." | |
| 19:36:28 | efried | okay, I need to go check whether you need to do something special the ram and disk (set them to zero) but I think those should be ignored. Meanwhile, the last of the three ^ things is to make sure there's no allocation present. | |
| 19:36:45 | efried | openstack resource provider allocation list (or show?) $node_uuid | |
| 19:37:34 | dking_desktop | That's empty. | |
| 19:37:53 | efried | Okay. Easier than me finding that code would be checking your placement logs. | |
| 19:38:25 | efried | Look for a line that has a GET call to the /allocation_candidates route with a querystring that includes CUSTOM_BAREMETAL_RESOURCE_CLASS | |
| 19:40:27 | efried | Okay, yeah, it looks like you need to set the flavor VCPUs to zero to make this work. | |
| 19:41:32 | efried | dking_desktop: like this: https://docs.openstack.org/ironic/latest/install/configure-nova-flavors | |
| 19:42:27 | efried | (not just VCPU, MEMORY_MB and DISK_GB too) | |
| 19:42:33 | dking_desktop | efried: Sorry, took me a minute to find it: http://paste.openstack.org/show/789547/ | |
| 19:42:52 | dking_desktop | Ah, so the VCPUs could be the problem? Let me see if I can update that. | |
| 19:43:58 | efried | Yup, so you see where that query is *also* asking for DISK_GB%3A1%2CMEMORY_MB%3A512%2CVCPU%3A1 (DISK_GB:1,MEMORY_MB:512,VCPU:1)? | |
| 19:44:13 | efried | your baremetal node's inventory doesn't have any of those resources. | |
| 19:44:46 | efried | Those are being fed in from your base flavor's disk/ram/vcpus | |
| 19:45:07 | efried | So the fix (ahem, it's a hack, I am ashamed) is to explicitly override those with zeros to take them out of the query. | |
| 19:46:02 | efried | And I think we did that hack because letting you set the base flavor values to real zeros would have blown up the code in a billion places | |
| 19:46:45 | efried | dking_desktop: anyway, if you follow https://docs.openstack.org/ironic/latest/install/configure-nova-flavors you should be able to make it work. | |
| 19:47:38 | dking_desktop | I'm having trouble getting it to accept 0. | |
| 19:48:25 | efried | dking_desktop: You can't set the base flavor properties to zero. You have to set *additional* extra specs for resources:{VCPU, MEMORY_MB, DISK_GB} | |
| 19:48:28 | efried | ... to zero | |
| 19:49:02 | efried | That will cause the scheduler to ignore the base vcpus/ram/disk values. | |
| 19:49:08 | efried | ...which can be set to whatever. | |
| 19:50:41 | dking_desktop | Ah. Yes, that's what's in the article, so that makes sense. Let me try that. | |
| 19:55:49 | dking_desktop | Great! That got much further. The build still failed, and I need to investigate that, but at least it started trying. | |
| 19:56:30 | efried | Okay, good deal. | |
| 19:59:16 | dking_desktop | I got: 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance cc78edb5-0268-4d71-a48d-61608b532d6f.' | |
| 19:59:35 | dking_desktop | Do you know where in the logs I might see where that failed? | |
| 20:01:37 | dking_desktop | Oh, I see it in nova-compute-ironic.log. It seems that the deploy image needs to be a UUID and not a name. | |
| 20:01:58 | efried | dking_desktop: I assume that was in your controller logs. You want to look in the compute log on the host that owns your ironic node to see why it bounced. | |
| 20:02:48 | dking_desktop | Yeah. nova-compute-ironic.log showed: Validation of image href deploy-initrd failed, reason: Scheme-less image href is not a UUID. | |
| 20:06:48 | dking_desktop | efried: Thank you very much for your help! I wouldn't have been able to make progress today without it. | |
| 20:07:11 | efried | dking_desktop: You're welcome. | |
| 20:08:02 | efried | dking_desktop: Reading between the lines, you're trying $old_release configurations/images/etc against $new_release code, and running into stuff we changed along the way. | |
| 20:09:37 | efried | dking_desktop: since it's ironic you're trying to deploy, you might also try the #openstack-ironic channel, as they'll generally be more familiar with the quirks in that area. dtantsur|afk I think would be particularly helpful, though he's Eastern Europe so best to hit him earlier in the day. | |
| 20:10:25 | dking_desktop | Earlier, I think I wasn't able to find any good documentation. My google searches kept taking me to outdated documentation, and so I think I started using that, only to find trouble down the road after I'd forgotten where I'd found the information. | |
| 20:10:39 | efried | ugh | |
| 20:11:08 | dking_desktop | And yes, the timezone issues make things difficult. I do most of my work while they're sleeping, though they're helpful early in my day. | |
| 20:11:26 | efried | well, what might work *sometimes* is, if you hit a page under docs.openstack.org/$proj/$rele/..., try replacing $rele with `latest` | |
| 20:11:59 | efried | You'll either get the latest instructions, or you'll get a 404, which means whatever you're trying to do won't work anyway :P | |
| 20:12:08 | efried | (in your case s/$rele/train/) | |
| 20:12:43 | efried | bauzas, sean-k-mooney, stephenfin: Responded on https://review.opendev.org/#/c/552924/ (NUMA RPs). | |
| 20:13:35 | efried | I would update it myself, but then stephenfin and gibi would have to be the approvers. | |
| 20:13:47 | efried | I guess we pretty much have to wait for morning anyway, so I'll just leave it. | |
| 20:15:13 | dking_desktop | Thank you so much. I think that now that I'm seeing the correct documentation, I may just start over and read each piece again in order. | |
| 20:16:11 | efried | dking_desktop: Cool. If in doing that you find discrepancies (or even typos), please propose edits. | |
| 20:16:37 | efried | ...or open bugs | |
| 20:16:47 | efried | ...or at least shout in here and we'll help you do ^ | |
| 20:17:19 | dking_desktop | Thanks. I submitted some a while back, and I've been the inspiration for a few more fixes already. Even if I'm having trouble, at least, there's updates being made somewhere. | |
| 20:30:21 | efried | Nova meeting in half an hour in #openstack-meeting. | |
| 20:30:21 | efried | Lots to discuss. Please plan to attend. | |
| 20:33:04 | sean-k-mooney | efried: sorry i was picking up the keys to my now house this afternoon. so i missed the spec discussions. im just getting dinner now but ill try and look at them in an hour or so. | |
| 20:33:18 | sean-k-mooney | i can try and attent the nova meeting too | |
| 20:35:29 | efried | sean-k-mooney: congrats on the house | |
| 20:36:02 | sean-k-mooney | now all i need is furniture, broadband, utilities and to move all my stuff :) | |
| 20:41:30 | efried | sean-k-mooney: would be nice to have a nova-side delegate who attended the vol local cache meeting. | |
| 20:41:56 | efried | I'm watching the replay, but I'm afraid it's not going to help me understand whether this is going to fly for U. | |
| 20:42:26 | efried | I assume lyarwood is sleeping? | |
| 20:43:58 | efried | sean-k-mooney: also, I added you to the os-vif release patch. Assume no reason not to merge that? https://review.opendev.org/707018 | |
| 21:10:13 | sean-k-mooney | not that i know of. i will check the review queue and +1 it soon | |
| 21:33:59 | sean-k-mooney | melwitt: it sound like the quota/flavor counting is boraderline a bug fix but ya if you think its better to do in V then thats fine | |
| 21:34:45 | sean-k-mooney | i say that because once we fix it upstream im sure we will be asked to backport it downstream | |
| 21:35:13 | melwitt | sean-k-mooney: normally it would be but I think because we need to leverage a new feature in placement and as part of that we'd have to migrate all of nova's allocations to have proper consumer types set, it's too big to be a bug fix. it's a spec | |
| 21:35:36 | sean-k-mooney | ah right ya that makes sense | |
| 21:41:17 | sean-k-mooney | i didnt think about the need to do a data migration of the allocations. | |
| 22:15:32 | efried | sean-k-mooney: you know that the num_implicit_numa_nodes thing is dead as of the latest rev, right? | |
| 22:15:44 | efried | so, your last comment is n/a? | |
| 22:17:38 | sean-k-mooney | i just noticed that now | |
| 22:17:45 | sean-k-mooney | so we are not doing the spliting? | |
| 22:18:08 | sean-k-mooney | i havent fully got up to speed with what changed sicne v19-21 | |
| 22:18:48 | sean-k-mooney | i see that the reporting is now a tristate True|false|none | |
| 22:19:06 | efried | right. But that's not explained at all in the doc; it needs to be. | |
| 22:19:59 | sean-k-mooney | are we still going to do the implict numa generation for non numa guests? or has that been removed too | |
| 22:22:23 | efried | removed | |
| 22:22:27 | efried | sean-k-mooney: we're back to segregating | |
| 22:22:50 | sean-k-mooney | ok | |
| 22:23:20 | sean-k-mooney | i might still propose my automatic asymetirc numa node change as a bug fix then | |
| 22:23:47 | sean-k-mooney | then someone can tell me its a feature and it can wait to V | |