Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-13
22:29:05 efried and it's not like you're sitting around trying them again and again to see if maybe they work now.
22:29:23 efried you would have tried them, gotten the bounce, and (if you understood the issue) just deleted them or modified them to work.
22:29:36 efried so this would just allow you to start making new flavors that aren't subject to that limitation.
22:29:45 dking_desktop I know this is the wrong place for this, but would anybody here happen to know, when a baremetal server is being deployed with "openstack server create ...", and it reboots to get a DHCP request, what service should be handling the DHCP request? I'm assuming that it's attempting to get the deploy_image from glance somehow.
22:29:46 sean-k-mooney well no so you can set them in the image
22:30:00 efried meh, same same.
22:30:30 sean-k-mooney efried: the point is the image may have had 5 cpus and you tried to use it with an image that asked for 2 numa nodes
22:30:42 efried oh, I see. Then it's not quite so simple.
22:30:55 sean-k-mooney ya so today it would fail
22:30:57 efried I didn't know you could ask for numa topo via the image. But that makes sense now I think about it.
22:31:01 sean-k-mooney with a tiny change it would work
22:31:14 dking_desktop sean-k-mooney: Would you happen to have any idea there? I think the folks over in #openstack-ironic are overseas and sleeping.
22:31:27 sean-k-mooney dking_desktop: the dhcp request is handeled by neutron dhcp agent
22:32:16 sean-k-mooney the way it works as part of the dhcp respoce we pass a dhcp option that tells the server where to find the ipxe image
22:32:26 sean-k-mooney that then deploys the ironic python agent
22:32:50 sean-k-mooney which connects to glance and streams the image onto the local disk of the ironic server
22:33:32 efried I'm out.
22:33:32 efried Good luck.
22:33:32 efried o/
22:33:45 dking_desktop efried: Have a great night!
22:34:19 sean-k-mooney dking_desktop: did that anser your question
22:34:50 sean-k-mooney by defualt the deploy image with the ironic python agent is served off a tftp share that is pxi booted not form glance
22:35:02 dking_desktop sean-k-mooney: Great! I suspected that. I'm looking at the neutron-dhcp-agent container. I see that it's running dnsmasq, but I can't see it listening anywhere.
22:35:17 sean-k-mooney dking_desktop: that said i know ironic have been working on redfish and http boot
22:35:36 sean-k-mooney dking_desktop: it will be running in a network namespace
22:35:56 dking_desktop I'd love to use Redfish. Unfortunately, I think there's a flaw in my server's redfish implimentation that causes it to fail when moving it to manage.
22:36:42 sean-k-mooney ya i have seen that altough i have only worked with prepoduction server that had redfish support so i was just happy it booted :)
22:39:12 dking_desktop I do like redfish, though. I'm using it for everything outside of openstack.
22:40:27 sean-k-mooney enginering samples of motherboads or alpha bios roms are not your friend whn trying to get redfish to work however
22:41:14 dking_desktop It seems that I'm not familiar with network namespaces. That's something new I suppose that I"ll need to learn about. I did find the configuration file, though. It only has one line, and that's for "log-facility".
22:41:37 sean-k-mooney dking_desktop: so anyway if you log into the network node and do "ip netns" you should see a bunch of network namespaces
22:42:19 sean-k-mooney the one where dnsmask is runnign will be dhcp_<network uuid> i think
22:42:39 dking_desktop There's just a couple of them at the moment, and one is the "qdhcp-..."
22:42:50 sean-k-mooney yep that is likely the one
22:43:03 sean-k-mooney q stands for quantum which is what neutron was originally called
22:44:04 sean-k-mooney so if you do "sudo ip netns exec qdhcp-.... bash" you will spawn a bash shell in the network namespace
22:44:26 sean-k-mooney then you can do "netstat -nlp"
22:44:36 sean-k-mooney and you shoudl see it listening on port 53?
22:44:42 sean-k-mooney that is the dhcp port right
22:45:12 dking_desktop I was thinking it was port 67. I think 53 is DNS.
22:45:22 dking_desktop But yes, both are there.
22:45:35 sean-k-mooney ah you are right 53 is dns
22:46:14 dking_desktop That's pretty neat. I see that I still have much to learn.
22:46:20 sean-k-mooney so if you install tcpdump or tskark you shoudl be able to dump the dhcp packets
22:46:55 sean-k-mooney i prefer tshark(the cli for wireshark) since it print the packets more nicely
22:47:19 sean-k-mooney so "tshark -i <interface> -V dhcp"
22:47:28 sean-k-mooney the -V is what prints the full packet
22:48:34 sean-k-mooney it might not recognise dhcpu in which case you would do 'tshark -i <interface> -V udp port 67 or 68'
22:49:37 dking_desktop I've been using tcpdump. So, I see that inside the network namespace, my devices are limited to just the loopback, and another, which I'm assuming is from an ovs bridge port.
22:49:50 sean-k-mooney yes
22:51:02 sean-k-mooney what is the actull issue you are having by the way
22:51:04 dking_desktop Is that for the provisioning_network?
22:51:50 sean-k-mooney so it depned on how you have it set up. i belive you can etiher use a seperate provisioning netwrok with a dnsmask manage by ironic or you can use a neutron netwrok
22:52:38 sean-k-mooney i should point out that i have not used ironic in about 4 release so they could have change things.
22:52:41 dking_desktop The issue is that I'm trying to deploy a baremetal server. Where I'm at currently is that I have created the baremetal node, introspected it, provided it, and I'm attempting to "openstack server create". I see that the node reboots and sends a DHCP request, but it gets no response, so it never completes the BUILD.
22:53:03 sean-k-mooney ah ok
22:53:13 sean-k-mooney is your provisioning network a neutron netwrok
22:53:26 sean-k-mooney if so did you make it a flat network
22:53:49 dking_desktop Yes, but I'm pretty sure I set it up incorrectly. I'm still trying to get familiar with openstack networking.
22:53:51 sean-k-mooney or are you using the external provioning network approch where the network is not manage by openstack
22:54:26 sean-k-mooney dking_desktop: i think the issue you are hitting is that ironic only optionally uses neutorn
22:54:51 sean-k-mooney in older release provioning was handeled by a non nuton network
22:55:01 sean-k-mooney in more recent release they use neutron
22:55:16 sean-k-mooney not all the docs are clear on whant you shoudl do in each case
22:56:38 sean-k-mooney i assume you are using enabled_network_interfaces=noop,flat,neutron and default_network_interface=neutron
22:56:44 dking_desktop I'm using train, currently. I'm open to whatever option works. I saw somewhere in the documentation that I should set cleaning_network, and then I got a complaint that I should set provisioning_network also. I didn't find any documentation, so I just made a flat network, and tried using that.
22:57:48 sean-k-mooney this is the relevent doc i think https://docs.openstack.org/ironic/train/install/configure-tenant-networks.html
22:58:12 dking_desktop enabled_network_interfaces = flat,neutron, and I don't have a default_network_interface.
22:58:24 sean-k-mooney dking_desktop: i think that is ok
22:59:09 sean-k-mooney its says if default_network_interface is not set the default network interface is determined by looking at the [dhcp]dhcp_provider
23:00:01 sean-k-mooney dking_desktop: did you disabel security groups for your provisioning and cleaning network
23:02:14 dking_desktop I was just reading about that. I did not.
23:02:40 dking_desktop I suppose that I should set cleaning_network_security_groups and provisioning_network_security_groups ? Are those the group names or IDs?
23:03:08 sean-k-mooney usually the uuid
23:04:10 sean-k-mooney if intospection is working then you are 90% of the way there
23:04:21 sean-k-mooney as that means 1 ironci can manage teh hardawer over ipmi
23:04:38 sean-k-mooney 2 it can serve the intospection ram disk
23:04:53 dking_desktop Yep. It took quite some time to get that working.
23:05:41 dking_desktop So, I know that it at least can get a ram disk to boot. I just have to figure out how to get the networking straight for provisioning.
23:07:06 sean-k-mooney ya unfortunetly i think you will have to ask either the ironic or neutron folks
23:07:29 sean-k-mooney i have done it years ago but i dont use ironic often so i have forgoten most of it
23:08:44 openstackgerrit Brian Rosmaita proposed openstack/nova master: Reject boot request for unsupported images https://review.opendev.org/707738
23:09:34 sean-k-mooney dking_desktop: do you need multi tenancy by the way for the provioning network
23:10:05 sean-k-mooney if its a private cloud you could look at teh simpler flat configuration
23:10:30 dking_desktop I might need to do that. Right now, I want to leave my options open.
23:11:37 dking_desktop Is the "provisioning_network" only to get the ramdisk booted and deploy the server? So, once that's done, it's either not necessary, or perhaps only for status updates?
23:12:03 sean-k-mooney yes basicaly
23:12:28 sean-k-mooney it is the network that need to have conectivity to where the image is located
23:12:40 sean-k-mooney and the tftp server
23:13:47 sean-k-mooney once the ironic node is provisioned it will normally use a different interface for the teant to ssh in/have netwrok conenctivity out onto the datacenter
23:15:31 sean-k-mooney dking_desktop: you might be hitting this by the way https://docs.openstack.org/ironic/train/admin/troubleshooting.html#dhcp-during-pxe-or-ipxe-is-inconsistent-or-unreliable
23:15:46 dking_desktop So, maybe you can help me here. Inside of the network namespace, I'm not seeing any DHCP requests. That explains why I didn't see anything logged and no responses.
23:16:41 sean-k-mooney ya so its possible the dhcp request is being droped by the switch before it gets to the contoler
23:17:23 dking_desktop So, how _should_ the packets be getting there? I see that this network interface is an ovs port inside of br-int. I know that br-int is patched to br-ex.
23:18:22 sean-k-mooney yes and the br-ex should have a physical interface attached
23:18:23 dking_desktop The server is booting up using DHCP/PXE, but it is on a trunked port, so the packets are coming in untagged. I know that's caused me trouble before.
23:18:49 sean-k-mooney right so if the neutron network is a flat netwrok
23:19:14 sean-k-mooney then it should be untag form the server, get to the top of rack switch and remain untagged
23:19:23 sean-k-mooney then as ita a broadcast it will flood
23:19:31 dking_desktop It does. It's attached to bond0. So, does br-ex send DHCP broadcasts to br-int, and then it sends them to all of its ports? That doesn't sound right.

Earlier   Later