Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-13
19:48:28 efried ... to zero
19:49:02 efried That will cause the scheduler to ignore the base vcpus/ram/disk values.
19:49:08 efried ...which can be set to whatever.
19:50:41 dking_desktop Ah. Yes, that's what's in the article, so that makes sense. Let me try that.
19:55:49 dking_desktop Great! That got much further. The build still failed, and I need to investigate that, but at least it started trying.
19:56:30 efried Okay, good deal.
19:59:16 dking_desktop I got: 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance cc78edb5-0268-4d71-a48d-61608b532d6f.'
19:59:35 dking_desktop Do you know where in the logs I might see where that failed?
20:01:37 dking_desktop Oh, I see it in nova-compute-ironic.log. It seems that the deploy image needs to be a UUID and not a name.
20:01:58 efried dking_desktop: I assume that was in your controller logs. You want to look in the compute log on the host that owns your ironic node to see why it bounced.
20:02:48 dking_desktop Yeah. nova-compute-ironic.log showed: Validation of image href deploy-initrd failed, reason: Scheme-less image href is not a UUID.
20:06:48 dking_desktop efried: Thank you very much for your help! I wouldn't have been able to make progress today without it.
20:07:11 efried dking_desktop: You're welcome.
20:08:02 efried dking_desktop: Reading between the lines, you're trying $old_release configurations/images/etc against $new_release code, and running into stuff we changed along the way.
20:09:37 efried dking_desktop: since it's ironic you're trying to deploy, you might also try the #openstack-ironic channel, as they'll generally be more familiar with the quirks in that area. dtantsur|afk I think would be particularly helpful, though he's Eastern Europe so best to hit him earlier in the day.
20:10:25 dking_desktop Earlier, I think I wasn't able to find any good documentation. My google searches kept taking me to outdated documentation, and so I think I started using that, only to find trouble down the road after I'd forgotten where I'd found the information.
20:10:39 efried ugh
20:11:08 dking_desktop And yes, the timezone issues make things difficult. I do most of my work while they're sleeping, though they're helpful early in my day.
20:11:26 efried well, what might work *sometimes* is, if you hit a page under docs.openstack.org/$proj/$rele/..., try replacing $rele with `latest`
20:11:59 efried You'll either get the latest instructions, or you'll get a 404, which means whatever you're trying to do won't work anyway :P
20:12:08 efried (in your case s/$rele/train/)
20:12:43 efried bauzas, sean-k-mooney, stephenfin: Responded on https://review.opendev.org/#/c/552924/ (NUMA RPs).
20:13:35 efried I would update it myself, but then stephenfin and gibi would have to be the approvers.
20:13:47 efried I guess we pretty much have to wait for morning anyway, so I'll just leave it.
20:15:13 dking_desktop Thank you so much. I think that now that I'm seeing the correct documentation, I may just start over and read each piece again in order.
20:16:11 efried dking_desktop: Cool. If in doing that you find discrepancies (or even typos), please propose edits.
20:16:37 efried ...or open bugs
20:16:47 efried ...or at least shout in here and we'll help you do ^
20:17:19 dking_desktop Thanks. I submitted some a while back, and I've been the inspiration for a few more fixes already. Even if I'm having trouble, at least, there's updates being made somewhere.
20:30:21 efried Nova meeting in half an hour in #openstack-meeting.
20:30:21 efried Lots to discuss. Please plan to attend.
20:33:04 sean-k-mooney efried: sorry i was picking up the keys to my now house this afternoon. so i missed the spec discussions. im just getting dinner now but ill try and look at them in an hour or so.
20:33:18 sean-k-mooney i can try and attent the nova meeting too
20:35:29 efried sean-k-mooney: congrats on the house
20:36:02 sean-k-mooney now all i need is furniture, broadband, utilities and to move all my stuff :)
20:41:30 efried sean-k-mooney: would be nice to have a nova-side delegate who attended the vol local cache meeting.
20:41:56 efried I'm watching the replay, but I'm afraid it's not going to help me understand whether this is going to fly for U.
20:42:26 efried I assume lyarwood is sleeping?
20:43:58 efried sean-k-mooney: also, I added you to the os-vif release patch. Assume no reason not to merge that? https://review.opendev.org/707018
21:10:13 sean-k-mooney not that i know of. i will check the review queue and +1 it soon
21:33:59 sean-k-mooney melwitt: it sound like the quota/flavor counting is boraderline a bug fix but ya if you think its better to do in V then thats fine
21:34:45 sean-k-mooney i say that because once we fix it upstream im sure we will be asked to backport it downstream
21:35:13 melwitt sean-k-mooney: normally it would be but I think because we need to leverage a new feature in placement and as part of that we'd have to migrate all of nova's allocations to have proper consumer types set, it's too big to be a bug fix. it's a spec
21:35:36 sean-k-mooney ah right ya that makes sense
21:41:17 sean-k-mooney i didnt think about the need to do a data migration of the allocations.
22:15:32 efried sean-k-mooney: you know that the num_implicit_numa_nodes thing is dead as of the latest rev, right?
22:15:44 efried so, your last comment is n/a?
22:17:38 sean-k-mooney i just noticed that now
22:17:45 sean-k-mooney so we are not doing the spliting?
22:18:08 sean-k-mooney i havent fully got up to speed with what changed sicne v19-21
22:18:48 sean-k-mooney i see that the reporting is now a tristate True|false|none
22:19:06 efried right. But that's not explained at all in the doc; it needs to be.
22:19:59 sean-k-mooney are we still going to do the implict numa generation for non numa guests? or has that been removed too
22:22:23 efried removed
22:22:27 efried sean-k-mooney: we're back to segregating
22:22:50 sean-k-mooney ok
22:23:20 sean-k-mooney i might still propose my automatic asymetirc numa node change as a bug fix then
22:23:47 sean-k-mooney then someone can tell me its a feature and it can wait to V
22:24:30 efried You mean:
22:24:30 efried Today if you say hw:numa_nodes=$x and we can't split $x evenly we bounce;
22:24:30 efried With your fix we would split asymmetrically, as close to evenly as possible
22:24:30 efried ?
22:24:46 sean-k-mooney yep
22:24:54 sean-k-mooney no other change
22:25:13 efried What's the error for that bounce today?
22:25:24 efried From the API, I imagine?
22:25:31 sean-k-mooney we have an exception we raise form the api yes
22:25:59 sean-k-mooney that says we cant generate an asemtic numa node configuration and you have to manally set it in the flavor/image
22:26:33 sean-k-mooney so it fails only at server boot but before we even create a build request
22:26:41 efried So yeah, I'm not sure if that counts as a bug fix or a feature. I'm also not sure whether it's important that it be discoverable or optional.
22:26:42 sean-k-mooney i think you will get a 400
22:26:50 efried I can see the argument either way.
22:27:54 sean-k-mooney its a triaval change too this fucntion https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1564-L1579
22:28:01 efried yeah, I understand the change
22:28:23 efried maybe gmann could weigh in as to whether it would need a microversion.
22:28:47 efried I would be okay without one, I think. Basically today if you have flavors that look like that they are useless.
22:29:02 sean-k-mooney it currently returns a 400 https://github.com/openstack/nova/blob/0d3aeb0287a0619695c9b9e17c2dec49099876a5/nova/exception.py#L1776
22:29:05 efried and it's not like you're sitting around trying them again and again to see if maybe they work now.
22:29:23 efried you would have tried them, gotten the bounce, and (if you understood the issue) just deleted them or modified them to work.
22:29:36 efried so this would just allow you to start making new flavors that aren't subject to that limitation.
22:29:45 dking_desktop I know this is the wrong place for this, but would anybody here happen to know, when a baremetal server is being deployed with "openstack server create ...", and it reboots to get a DHCP request, what service should be handling the DHCP request? I'm assuming that it's attempting to get the deploy_image from glance somehow.
22:29:46 sean-k-mooney well no so you can set them in the image
22:30:00 efried meh, same same.
22:30:30 sean-k-mooney efried: the point is the image may have had 5 cpus and you tried to use it with an image that asked for 2 numa nodes
22:30:42 efried oh, I see. Then it's not quite so simple.
22:30:55 sean-k-mooney ya so today it would fail
22:30:57 efried I didn't know you could ask for numa topo via the image. But that makes sense now I think about it.
22:31:01 sean-k-mooney with a tiny change it would work
22:31:14 dking_desktop sean-k-mooney: Would you happen to have any idea there? I think the folks over in #openstack-ironic are overseas and sleeping.
22:31:27 sean-k-mooney dking_desktop: the dhcp request is handeled by neutron dhcp agent
22:32:16 sean-k-mooney the way it works as part of the dhcp respoce we pass a dhcp option that tells the server where to find the ipxe image
22:32:26 sean-k-mooney that then deploys the ironic python agent
22:32:50 sean-k-mooney which connects to glance and streams the image onto the local disk of the ironic server
22:33:32 efried I'm out.
22:33:32 efried Good luck.
22:33:32 efried o/
22:33:45 dking_desktop efried: Have a great night!
22:34:19 sean-k-mooney dking_desktop: did that anser your question
22:34:50 sean-k-mooney by defualt the deploy image with the ironic python agent is served off a tftp share that is pxi booted not form glance
22:35:02 dking_desktop sean-k-mooney: Great! I suspected that. I'm looking at the neutron-dhcp-agent container. I see that it's running dnsmasq, but I can't see it listening anywhere.

Earlier   Later