| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 19:37:53 | efried | Okay. Easier than me finding that code would be checking your placement logs. | |
| 19:38:25 | efried | Look for a line that has a GET call to the /allocation_candidates route with a querystring that includes CUSTOM_BAREMETAL_RESOURCE_CLASS | |
| 19:40:27 | efried | Okay, yeah, it looks like you need to set the flavor VCPUs to zero to make this work. | |
| 19:41:32 | efried | dking_desktop: like this: https://docs.openstack.org/ironic/latest/install/configure-nova-flavors | |
| 19:42:27 | efried | (not just VCPU, MEMORY_MB and DISK_GB too) | |
| 19:42:33 | dking_desktop | efried: Sorry, took me a minute to find it: http://paste.openstack.org/show/789547/ | |
| 19:42:52 | dking_desktop | Ah, so the VCPUs could be the problem? Let me see if I can update that. | |
| 19:43:58 | efried | Yup, so you see where that query is *also* asking for DISK_GB%3A1%2CMEMORY_MB%3A512%2CVCPU%3A1 (DISK_GB:1,MEMORY_MB:512,VCPU:1)? | |
| 19:44:13 | efried | your baremetal node's inventory doesn't have any of those resources. | |
| 19:44:46 | efried | Those are being fed in from your base flavor's disk/ram/vcpus | |
| 19:45:07 | efried | So the fix (ahem, it's a hack, I am ashamed) is to explicitly override those with zeros to take them out of the query. | |
| 19:46:02 | efried | And I think we did that hack because letting you set the base flavor values to real zeros would have blown up the code in a billion places | |
| 19:46:45 | efried | dking_desktop: anyway, if you follow https://docs.openstack.org/ironic/latest/install/configure-nova-flavors you should be able to make it work. | |
| 19:47:38 | dking_desktop | I'm having trouble getting it to accept 0. | |
| 19:48:25 | efried | dking_desktop: You can't set the base flavor properties to zero. You have to set *additional* extra specs for resources:{VCPU, MEMORY_MB, DISK_GB} | |
| 19:48:28 | efried | ... to zero | |
| 19:49:02 | efried | That will cause the scheduler to ignore the base vcpus/ram/disk values. | |
| 19:49:08 | efried | ...which can be set to whatever. | |
| 19:50:41 | dking_desktop | Ah. Yes, that's what's in the article, so that makes sense. Let me try that. | |
| 19:55:49 | dking_desktop | Great! That got much further. The build still failed, and I need to investigate that, but at least it started trying. | |
| 19:56:30 | efried | Okay, good deal. | |
| 19:59:16 | dking_desktop | I got: 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance cc78edb5-0268-4d71-a48d-61608b532d6f.' | |
| 19:59:35 | dking_desktop | Do you know where in the logs I might see where that failed? | |
| 20:01:37 | dking_desktop | Oh, I see it in nova-compute-ironic.log. It seems that the deploy image needs to be a UUID and not a name. | |
| 20:01:58 | efried | dking_desktop: I assume that was in your controller logs. You want to look in the compute log on the host that owns your ironic node to see why it bounced. | |
| 20:02:48 | dking_desktop | Yeah. nova-compute-ironic.log showed: Validation of image href deploy-initrd failed, reason: Scheme-less image href is not a UUID. | |
| 20:06:48 | dking_desktop | efried: Thank you very much for your help! I wouldn't have been able to make progress today without it. | |
| 20:07:11 | efried | dking_desktop: You're welcome. | |
| 20:08:02 | efried | dking_desktop: Reading between the lines, you're trying $old_release configurations/images/etc against $new_release code, and running into stuff we changed along the way. | |
| 20:09:37 | efried | dking_desktop: since it's ironic you're trying to deploy, you might also try the #openstack-ironic channel, as they'll generally be more familiar with the quirks in that area. dtantsur|afk I think would be particularly helpful, though he's Eastern Europe so best to hit him earlier in the day. | |
| 20:10:25 | dking_desktop | Earlier, I think I wasn't able to find any good documentation. My google searches kept taking me to outdated documentation, and so I think I started using that, only to find trouble down the road after I'd forgotten where I'd found the information. | |
| 20:10:39 | efried | ugh | |
| 20:11:08 | dking_desktop | And yes, the timezone issues make things difficult. I do most of my work while they're sleeping, though they're helpful early in my day. | |
| 20:11:26 | efried | well, what might work *sometimes* is, if you hit a page under docs.openstack.org/$proj/$rele/..., try replacing $rele with `latest` | |
| 20:11:59 | efried | You'll either get the latest instructions, or you'll get a 404, which means whatever you're trying to do won't work anyway :P | |
| 20:12:08 | efried | (in your case s/$rele/train/) | |
| 20:12:43 | efried | bauzas, sean-k-mooney, stephenfin: Responded on https://review.opendev.org/#/c/552924/ (NUMA RPs). | |
| 20:13:35 | efried | I would update it myself, but then stephenfin and gibi would have to be the approvers. | |
| 20:13:47 | efried | I guess we pretty much have to wait for morning anyway, so I'll just leave it. | |
| 20:15:13 | dking_desktop | Thank you so much. I think that now that I'm seeing the correct documentation, I may just start over and read each piece again in order. | |
| 20:16:11 | efried | dking_desktop: Cool. If in doing that you find discrepancies (or even typos), please propose edits. | |
| 20:16:37 | efried | ...or open bugs | |
| 20:16:47 | efried | ...or at least shout in here and we'll help you do ^ | |
| 20:17:19 | dking_desktop | Thanks. I submitted some a while back, and I've been the inspiration for a few more fixes already. Even if I'm having trouble, at least, there's updates being made somewhere. | |
| 20:30:21 | efried | Lots to discuss. Please plan to attend. | |
| 20:30:21 | efried | Nova meeting in half an hour in #openstack-meeting. | |
| 20:33:04 | sean-k-mooney | efried: sorry i was picking up the keys to my now house this afternoon. so i missed the spec discussions. im just getting dinner now but ill try and look at them in an hour or so. | |
| 20:33:18 | sean-k-mooney | i can try and attent the nova meeting too | |
| 20:35:29 | efried | sean-k-mooney: congrats on the house | |
| 20:36:02 | sean-k-mooney | now all i need is furniture, broadband, utilities and to move all my stuff :) | |
| 20:41:30 | efried | sean-k-mooney: would be nice to have a nova-side delegate who attended the vol local cache meeting. | |
| 20:41:56 | efried | I'm watching the replay, but I'm afraid it's not going to help me understand whether this is going to fly for U. | |
| 20:42:26 | efried | I assume lyarwood is sleeping? | |
| 20:43:58 | efried | sean-k-mooney: also, I added you to the os-vif release patch. Assume no reason not to merge that? https://review.opendev.org/707018 | |
| 21:10:13 | sean-k-mooney | not that i know of. i will check the review queue and +1 it soon | |
| 21:33:59 | sean-k-mooney | melwitt: it sound like the quota/flavor counting is boraderline a bug fix but ya if you think its better to do in V then thats fine | |
| 21:34:45 | sean-k-mooney | i say that because once we fix it upstream im sure we will be asked to backport it downstream | |
| 21:35:13 | melwitt | sean-k-mooney: normally it would be but I think because we need to leverage a new feature in placement and as part of that we'd have to migrate all of nova's allocations to have proper consumer types set, it's too big to be a bug fix. it's a spec | |
| 21:35:36 | sean-k-mooney | ah right ya that makes sense | |
| 21:41:17 | sean-k-mooney | i didnt think about the need to do a data migration of the allocations. | |
| 22:15:32 | efried | sean-k-mooney: you know that the num_implicit_numa_nodes thing is dead as of the latest rev, right? | |
| 22:15:44 | efried | so, your last comment is n/a? | |
| 22:17:38 | sean-k-mooney | i just noticed that now | |
| 22:17:45 | sean-k-mooney | so we are not doing the spliting? | |
| 22:18:08 | sean-k-mooney | i havent fully got up to speed with what changed sicne v19-21 | |
| 22:18:48 | sean-k-mooney | i see that the reporting is now a tristate True|false|none | |
| 22:19:06 | efried | right. But that's not explained at all in the doc; it needs to be. | |
| 22:19:59 | sean-k-mooney | are we still going to do the implict numa generation for non numa guests? or has that been removed too | |
| 22:22:23 | efried | removed | |
| 22:22:27 | efried | sean-k-mooney: we're back to segregating | |
| 22:22:50 | sean-k-mooney | ok | |
| 22:23:20 | sean-k-mooney | i might still propose my automatic asymetirc numa node change as a bug fix then | |
| 22:23:47 | sean-k-mooney | then someone can tell me its a feature and it can wait to V | |
| 22:24:30 | efried | ? | |
| 22:24:30 | efried | With your fix we would split asymmetrically, as close to evenly as possible | |
| 22:24:30 | efried | Today if you say hw:numa_nodes=$x and we can't split $x evenly we bounce; | |
| 22:24:30 | efried | You mean: | |
| 22:24:46 | sean-k-mooney | yep | |
| 22:24:54 | sean-k-mooney | no other change | |
| 22:25:13 | efried | What's the error for that bounce today? | |
| 22:25:24 | efried | From the API, I imagine? | |
| 22:25:31 | sean-k-mooney | we have an exception we raise form the api yes | |
| 22:25:59 | sean-k-mooney | that says we cant generate an asemtic numa node configuration and you have to manally set it in the flavor/image | |
| 22:26:33 | sean-k-mooney | so it fails only at server boot but before we even create a build request | |
| 22:26:41 | efried | So yeah, I'm not sure if that counts as a bug fix or a feature. I'm also not sure whether it's important that it be discoverable or optional. | |
| 22:26:42 | sean-k-mooney | i think you will get a 400 | |
| 22:26:50 | efried | I can see the argument either way. | |
| 22:27:54 | sean-k-mooney | its a triaval change too this fucntion https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1564-L1579 | |
| 22:28:01 | efried | yeah, I understand the change | |
| 22:28:23 | efried | maybe gmann could weigh in as to whether it would need a microversion. | |
| 22:28:47 | efried | I would be okay without one, I think. Basically today if you have flavors that look like that they are useless. | |
| 22:29:02 | sean-k-mooney | it currently returns a 400 https://github.com/openstack/nova/blob/0d3aeb0287a0619695c9b9e17c2dec49099876a5/nova/exception.py#L1776 | |
| 22:29:05 | efried | and it's not like you're sitting around trying them again and again to see if maybe they work now. | |
| 22:29:23 | efried | you would have tried them, gotten the bounce, and (if you understood the issue) just deleted them or modified them to work. | |
| 22:29:36 | efried | so this would just allow you to start making new flavors that aren't subject to that limitation. | |
| 22:29:45 | dking_desktop | I know this is the wrong place for this, but would anybody here happen to know, when a baremetal server is being deployed with "openstack server create ...", and it reboots to get a DHCP request, what service should be handling the DHCP request? I'm assuming that it's attempting to get the deploy_image from glance somehow. | |
| 22:29:46 | sean-k-mooney | well no so you can set them in the image | |
| 22:30:00 | efried | meh, same same. | |
| 22:30:30 | sean-k-mooney | efried: the point is the image may have had 5 cpus and you tried to use it with an image that asked for 2 numa nodes | |
| 22:30:42 | efried | oh, I see. Then it's not quite so simple. | |