| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 19:31:28 | efried | should show you that. | |
| 19:31:28 | efried | openstack resource provider inventory list (or maybe show) $node_uuid | |
| 19:31:28 | efried | something like | |
| 19:31:38 | dking_desktop | I'm using train. | |
| 19:32:45 | efried | Then you won't have to add --os-placement-api-version with every command. | |
| 19:32:45 | efried | (I think I'm spelling that var name right) | |
| 19:32:45 | efried | export OS_PLACEMENT_API_VERSION=1.36 | |
| 19:32:45 | efried | okay, so you might want to make life easier with | |
| 19:32:50 | dking_desktop | Okay, that shows me the resource class and a few other pieces of info: http://paste.openstack.org/show/789545/ | |
| 19:34:06 | efried | Great. So the flavor you're using should be asking for resources:CUSTOM_BAREMETAL_RESOURCE_CLASS=1 | |
| 19:34:08 | efried | is it? | |
| 19:35:05 | dking_desktop | Yes, it is: http://paste.openstack.org/show/789546/ | |
| 19:35:18 | efried | the fact that your trait list didn't show COMPUTE_STATUS_DISABLED, and your inventory showed reserved=0, is two out of the three things that should make this node eligible for scheduling. | |
| 19:36:17 | dking_desktop | That sounds good. Any idea why I would be getting "No valid host was found." | |
| 19:36:28 | efried | okay, I need to go check whether you need to do something special the ram and disk (set them to zero) but I think those should be ignored. Meanwhile, the last of the three ^ things is to make sure there's no allocation present. | |
| 19:36:45 | efried | openstack resource provider allocation list (or show?) $node_uuid | |
| 19:37:34 | dking_desktop | That's empty. | |
| 19:37:53 | efried | Okay. Easier than me finding that code would be checking your placement logs. | |
| 19:38:25 | efried | Look for a line that has a GET call to the /allocation_candidates route with a querystring that includes CUSTOM_BAREMETAL_RESOURCE_CLASS | |
| 19:40:27 | efried | Okay, yeah, it looks like you need to set the flavor VCPUs to zero to make this work. | |
| 19:41:32 | efried | dking_desktop: like this: https://docs.openstack.org/ironic/latest/install/configure-nova-flavors | |
| 19:42:27 | efried | (not just VCPU, MEMORY_MB and DISK_GB too) | |
| 19:42:33 | dking_desktop | efried: Sorry, took me a minute to find it: http://paste.openstack.org/show/789547/ | |
| 19:42:52 | dking_desktop | Ah, so the VCPUs could be the problem? Let me see if I can update that. | |
| 19:43:58 | efried | Yup, so you see where that query is *also* asking for DISK_GB%3A1%2CMEMORY_MB%3A512%2CVCPU%3A1 (DISK_GB:1,MEMORY_MB:512,VCPU:1)? | |
| 19:44:13 | efried | your baremetal node's inventory doesn't have any of those resources. | |
| 19:44:46 | efried | Those are being fed in from your base flavor's disk/ram/vcpus | |
| 19:45:07 | efried | So the fix (ahem, it's a hack, I am ashamed) is to explicitly override those with zeros to take them out of the query. | |
| 19:46:02 | efried | And I think we did that hack because letting you set the base flavor values to real zeros would have blown up the code in a billion places | |
| 19:46:45 | efried | dking_desktop: anyway, if you follow https://docs.openstack.org/ironic/latest/install/configure-nova-flavors you should be able to make it work. | |
| 19:47:38 | dking_desktop | I'm having trouble getting it to accept 0. | |
| 19:48:25 | efried | dking_desktop: You can't set the base flavor properties to zero. You have to set *additional* extra specs for resources:{VCPU, MEMORY_MB, DISK_GB} | |
| 19:48:28 | efried | ... to zero | |
| 19:49:02 | efried | That will cause the scheduler to ignore the base vcpus/ram/disk values. | |
| 19:49:08 | efried | ...which can be set to whatever. | |
| 19:50:41 | dking_desktop | Ah. Yes, that's what's in the article, so that makes sense. Let me try that. | |
| 19:55:49 | dking_desktop | Great! That got much further. The build still failed, and I need to investigate that, but at least it started trying. | |
| 19:56:30 | efried | Okay, good deal. | |
| 19:59:16 | dking_desktop | I got: 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance cc78edb5-0268-4d71-a48d-61608b532d6f.' | |
| 19:59:35 | dking_desktop | Do you know where in the logs I might see where that failed? | |
| 20:01:37 | dking_desktop | Oh, I see it in nova-compute-ironic.log. It seems that the deploy image needs to be a UUID and not a name. | |
| 20:01:58 | efried | dking_desktop: I assume that was in your controller logs. You want to look in the compute log on the host that owns your ironic node to see why it bounced. | |
| 20:02:48 | dking_desktop | Yeah. nova-compute-ironic.log showed: Validation of image href deploy-initrd failed, reason: Scheme-less image href is not a UUID. | |
| 20:06:48 | dking_desktop | efried: Thank you very much for your help! I wouldn't have been able to make progress today without it. | |
| 20:07:11 | efried | dking_desktop: You're welcome. | |
| 20:08:02 | efried | dking_desktop: Reading between the lines, you're trying $old_release configurations/images/etc against $new_release code, and running into stuff we changed along the way. | |
| 20:09:37 | efried | dking_desktop: since it's ironic you're trying to deploy, you might also try the #openstack-ironic channel, as they'll generally be more familiar with the quirks in that area. dtantsur|afk I think would be particularly helpful, though he's Eastern Europe so best to hit him earlier in the day. | |
| 20:10:25 | dking_desktop | Earlier, I think I wasn't able to find any good documentation. My google searches kept taking me to outdated documentation, and so I think I started using that, only to find trouble down the road after I'd forgotten where I'd found the information. | |
| 20:10:39 | efried | ugh | |
| 20:11:08 | dking_desktop | And yes, the timezone issues make things difficult. I do most of my work while they're sleeping, though they're helpful early in my day. | |
| 20:11:26 | efried | well, what might work *sometimes* is, if you hit a page under docs.openstack.org/$proj/$rele/..., try replacing $rele with `latest` | |
| 20:11:59 | efried | You'll either get the latest instructions, or you'll get a 404, which means whatever you're trying to do won't work anyway :P | |
| 20:12:08 | efried | (in your case s/$rele/train/) | |
| 20:12:43 | efried | bauzas, sean-k-mooney, stephenfin: Responded on https://review.opendev.org/#/c/552924/ (NUMA RPs). | |
| 20:13:35 | efried | I would update it myself, but then stephenfin and gibi would have to be the approvers. | |
| 20:13:47 | efried | I guess we pretty much have to wait for morning anyway, so I'll just leave it. | |
| 20:15:13 | dking_desktop | Thank you so much. I think that now that I'm seeing the correct documentation, I may just start over and read each piece again in order. | |
| 20:16:11 | efried | dking_desktop: Cool. If in doing that you find discrepancies (or even typos), please propose edits. | |
| 20:16:37 | efried | ...or open bugs | |
| 20:16:47 | efried | ...or at least shout in here and we'll help you do ^ | |
| 20:17:19 | dking_desktop | Thanks. I submitted some a while back, and I've been the inspiration for a few more fixes already. Even if I'm having trouble, at least, there's updates being made somewhere. | |
| 20:30:21 | efried | Lots to discuss. Please plan to attend. | |
| 20:30:21 | efried | Nova meeting in half an hour in #openstack-meeting. | |
| 20:33:04 | sean-k-mooney | efried: sorry i was picking up the keys to my now house this afternoon. so i missed the spec discussions. im just getting dinner now but ill try and look at them in an hour or so. | |
| 20:33:18 | sean-k-mooney | i can try and attent the nova meeting too | |
| 20:35:29 | efried | sean-k-mooney: congrats on the house | |
| 20:36:02 | sean-k-mooney | now all i need is furniture, broadband, utilities and to move all my stuff :) | |
| 20:41:30 | efried | sean-k-mooney: would be nice to have a nova-side delegate who attended the vol local cache meeting. | |
| 20:41:56 | efried | I'm watching the replay, but I'm afraid it's not going to help me understand whether this is going to fly for U. | |
| 20:42:26 | efried | I assume lyarwood is sleeping? | |
| 20:43:58 | efried | sean-k-mooney: also, I added you to the os-vif release patch. Assume no reason not to merge that? https://review.opendev.org/707018 | |
| 21:10:13 | sean-k-mooney | not that i know of. i will check the review queue and +1 it soon | |
| 21:33:59 | sean-k-mooney | melwitt: it sound like the quota/flavor counting is boraderline a bug fix but ya if you think its better to do in V then thats fine | |
| 21:34:45 | sean-k-mooney | i say that because once we fix it upstream im sure we will be asked to backport it downstream | |
| 21:35:13 | melwitt | sean-k-mooney: normally it would be but I think because we need to leverage a new feature in placement and as part of that we'd have to migrate all of nova's allocations to have proper consumer types set, it's too big to be a bug fix. it's a spec | |
| 21:35:36 | sean-k-mooney | ah right ya that makes sense | |
| 21:41:17 | sean-k-mooney | i didnt think about the need to do a data migration of the allocations. | |
| 22:15:32 | efried | sean-k-mooney: you know that the num_implicit_numa_nodes thing is dead as of the latest rev, right? | |
| 22:15:44 | efried | so, your last comment is n/a? | |
| 22:17:38 | sean-k-mooney | i just noticed that now | |
| 22:17:45 | sean-k-mooney | so we are not doing the spliting? | |
| 22:18:08 | sean-k-mooney | i havent fully got up to speed with what changed sicne v19-21 | |
| 22:18:48 | sean-k-mooney | i see that the reporting is now a tristate True|false|none | |
| 22:19:06 | efried | right. But that's not explained at all in the doc; it needs to be. | |
| 22:19:59 | sean-k-mooney | are we still going to do the implict numa generation for non numa guests? or has that been removed too | |
| 22:22:23 | efried | removed | |
| 22:22:27 | efried | sean-k-mooney: we're back to segregating | |
| 22:22:50 | sean-k-mooney | ok | |
| 22:23:20 | sean-k-mooney | i might still propose my automatic asymetirc numa node change as a bug fix then | |
| 22:23:47 | sean-k-mooney | then someone can tell me its a feature and it can wait to V | |
| 22:24:30 | efried | ? | |
| 22:24:30 | efried | With your fix we would split asymmetrically, as close to evenly as possible | |
| 22:24:30 | efried | Today if you say hw:numa_nodes=$x and we can't split $x evenly we bounce; | |
| 22:24:30 | efried | You mean: | |
| 22:24:46 | sean-k-mooney | yep | |
| 22:24:54 | sean-k-mooney | no other change | |
| 22:25:13 | efried | What's the error for that bounce today? | |
| 22:25:24 | efried | From the API, I imagine? | |
| 22:25:31 | sean-k-mooney | we have an exception we raise form the api yes | |
| 22:25:59 | sean-k-mooney | that says we cant generate an asemtic numa node configuration and you have to manally set it in the flavor/image | |