| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 22:35:36 | sean-k-mooney | dking_desktop: it will be running in a network namespace | |
| 22:35:56 | dking_desktop | I'd love to use Redfish. Unfortunately, I think there's a flaw in my server's redfish implimentation that causes it to fail when moving it to manage. | |
| 22:36:42 | sean-k-mooney | ya i have seen that altough i have only worked with prepoduction server that had redfish support so i was just happy it booted :) | |
| 22:39:12 | dking_desktop | I do like redfish, though. I'm using it for everything outside of openstack. | |
| 22:40:27 | sean-k-mooney | enginering samples of motherboads or alpha bios roms are not your friend whn trying to get redfish to work however | |
| 22:41:14 | dking_desktop | It seems that I'm not familiar with network namespaces. That's something new I suppose that I"ll need to learn about. I did find the configuration file, though. It only has one line, and that's for "log-facility". | |
| 22:41:37 | sean-k-mooney | dking_desktop: so anyway if you log into the network node and do "ip netns" you should see a bunch of network namespaces | |
| 22:42:19 | sean-k-mooney | the one where dnsmask is runnign will be dhcp_<network uuid> i think | |
| 22:42:39 | dking_desktop | There's just a couple of them at the moment, and one is the "qdhcp-..." | |
| 22:42:50 | sean-k-mooney | yep that is likely the one | |
| 22:43:03 | sean-k-mooney | q stands for quantum which is what neutron was originally called | |
| 22:44:04 | sean-k-mooney | so if you do "sudo ip netns exec qdhcp-.... bash" you will spawn a bash shell in the network namespace | |
| 22:44:26 | sean-k-mooney | then you can do "netstat -nlp" | |
| 22:44:36 | sean-k-mooney | and you shoudl see it listening on port 53? | |
| 22:44:42 | sean-k-mooney | that is the dhcp port right | |
| 22:45:12 | dking_desktop | I was thinking it was port 67. I think 53 is DNS. | |
| 22:45:22 | dking_desktop | But yes, both are there. | |
| 22:45:35 | sean-k-mooney | ah you are right 53 is dns | |
| 22:46:14 | dking_desktop | That's pretty neat. I see that I still have much to learn. | |
| 22:46:20 | sean-k-mooney | so if you install tcpdump or tskark you shoudl be able to dump the dhcp packets | |
| 22:46:55 | sean-k-mooney | i prefer tshark(the cli for wireshark) since it print the packets more nicely | |
| 22:47:19 | sean-k-mooney | so "tshark -i <interface> -V dhcp" | |
| 22:47:28 | sean-k-mooney | the -V is what prints the full packet | |
| 22:48:34 | sean-k-mooney | it might not recognise dhcpu in which case you would do 'tshark -i <interface> -V udp port 67 or 68' | |
| 22:49:37 | dking_desktop | I've been using tcpdump. So, I see that inside the network namespace, my devices are limited to just the loopback, and another, which I'm assuming is from an ovs bridge port. | |
| 22:49:50 | sean-k-mooney | yes | |
| 22:51:02 | sean-k-mooney | what is the actull issue you are having by the way | |
| 22:51:04 | dking_desktop | Is that for the provisioning_network? | |
| 22:51:50 | sean-k-mooney | so it depned on how you have it set up. i belive you can etiher use a seperate provisioning netwrok with a dnsmask manage by ironic or you can use a neutron netwrok | |
| 22:52:38 | sean-k-mooney | i should point out that i have not used ironic in about 4 release so they could have change things. | |
| 22:52:41 | dking_desktop | The issue is that I'm trying to deploy a baremetal server. Where I'm at currently is that I have created the baremetal node, introspected it, provided it, and I'm attempting to "openstack server create". I see that the node reboots and sends a DHCP request, but it gets no response, so it never completes the BUILD. | |
| 22:53:03 | sean-k-mooney | ah ok | |
| 22:53:13 | sean-k-mooney | is your provisioning network a neutron netwrok | |
| 22:53:26 | sean-k-mooney | if so did you make it a flat network | |
| 22:53:49 | dking_desktop | Yes, but I'm pretty sure I set it up incorrectly. I'm still trying to get familiar with openstack networking. | |
| 22:53:51 | sean-k-mooney | or are you using the external provioning network approch where the network is not manage by openstack | |
| 22:54:26 | sean-k-mooney | dking_desktop: i think the issue you are hitting is that ironic only optionally uses neutorn | |
| 22:54:51 | sean-k-mooney | in older release provioning was handeled by a non nuton network | |
| 22:55:01 | sean-k-mooney | in more recent release they use neutron | |
| 22:55:16 | sean-k-mooney | not all the docs are clear on whant you shoudl do in each case | |
| 22:56:38 | sean-k-mooney | i assume you are using enabled_network_interfaces=noop,flat,neutron and default_network_interface=neutron | |
| 22:56:44 | dking_desktop | I'm using train, currently. I'm open to whatever option works. I saw somewhere in the documentation that I should set cleaning_network, and then I got a complaint that I should set provisioning_network also. I didn't find any documentation, so I just made a flat network, and tried using that. | |
| 22:57:48 | sean-k-mooney | this is the relevent doc i think https://docs.openstack.org/ironic/train/install/configure-tenant-networks.html | |
| 22:58:12 | dking_desktop | enabled_network_interfaces = flat,neutron, and I don't have a default_network_interface. | |
| 22:58:24 | sean-k-mooney | dking_desktop: i think that is ok | |
| 22:59:09 | sean-k-mooney | its says if default_network_interface is not set the default network interface is determined by looking at the [dhcp]dhcp_provider | |
| 23:00:01 | sean-k-mooney | dking_desktop: did you disabel security groups for your provisioning and cleaning network | |
| 23:02:14 | dking_desktop | I was just reading about that. I did not. | |
| 23:02:40 | dking_desktop | I suppose that I should set cleaning_network_security_groups and provisioning_network_security_groups ? Are those the group names or IDs? | |
| 23:03:08 | sean-k-mooney | usually the uuid | |
| 23:04:10 | sean-k-mooney | if intospection is working then you are 90% of the way there | |
| 23:04:21 | sean-k-mooney | as that means 1 ironci can manage teh hardawer over ipmi | |
| 23:04:38 | sean-k-mooney | 2 it can serve the intospection ram disk | |
| 23:04:53 | dking_desktop | Yep. It took quite some time to get that working. | |
| 23:05:41 | dking_desktop | So, I know that it at least can get a ram disk to boot. I just have to figure out how to get the networking straight for provisioning. | |
| 23:07:06 | sean-k-mooney | ya unfortunetly i think you will have to ask either the ironic or neutron folks | |
| 23:07:29 | sean-k-mooney | i have done it years ago but i dont use ironic often so i have forgoten most of it | |
| 23:08:44 | openstackgerrit | Brian Rosmaita proposed openstack/nova master: Reject boot request for unsupported images https://review.opendev.org/707738 | |
| 23:09:34 | sean-k-mooney | dking_desktop: do you need multi tenancy by the way for the provioning network | |
| 23:10:05 | sean-k-mooney | if its a private cloud you could look at teh simpler flat configuration | |
| 23:10:30 | dking_desktop | I might need to do that. Right now, I want to leave my options open. | |
| 23:11:37 | dking_desktop | Is the "provisioning_network" only to get the ramdisk booted and deploy the server? So, once that's done, it's either not necessary, or perhaps only for status updates? | |
| 23:12:03 | sean-k-mooney | yes basicaly | |
| 23:12:28 | sean-k-mooney | it is the network that need to have conectivity to where the image is located | |
| 23:12:40 | sean-k-mooney | and the tftp server | |
| 23:13:47 | sean-k-mooney | once the ironic node is provisioned it will normally use a different interface for the teant to ssh in/have netwrok conenctivity out onto the datacenter | |
| 23:15:31 | sean-k-mooney | dking_desktop: you might be hitting this by the way https://docs.openstack.org/ironic/train/admin/troubleshooting.html#dhcp-during-pxe-or-ipxe-is-inconsistent-or-unreliable | |
| 23:15:46 | dking_desktop | So, maybe you can help me here. Inside of the network namespace, I'm not seeing any DHCP requests. That explains why I didn't see anything logged and no responses. | |
| 23:16:41 | sean-k-mooney | ya so its possible the dhcp request is being droped by the switch before it gets to the contoler | |
| 23:17:23 | dking_desktop | So, how _should_ the packets be getting there? I see that this network interface is an ovs port inside of br-int. I know that br-int is patched to br-ex. | |
| 23:18:22 | sean-k-mooney | yes and the br-ex should have a physical interface attached | |
| 23:18:23 | dking_desktop | The server is booting up using DHCP/PXE, but it is on a trunked port, so the packets are coming in untagged. I know that's caused me trouble before. | |
| 23:18:49 | sean-k-mooney | right so if the neutron network is a flat netwrok | |
| 23:19:14 | sean-k-mooney | then it should be untag form the server, get to the top of rack switch and remain untagged | |
| 23:19:23 | sean-k-mooney | then as ita a broadcast it will flood | |
| 23:19:31 | dking_desktop | It does. It's attached to bond0. So, does br-ex send DHCP broadcasts to br-int, and then it sends them to all of its ports? That doesn't sound right. | |
| 23:19:31 | sean-k-mooney | eventually make it to the contoler | |
| 23:20:23 | sean-k-mooney | when it arrives in the contoler it will enter the br-ex. it will be vlan taged with a local vlan and then be flooded to only the ports fo that vlan | |
| 23:20:39 | sean-k-mooney | then it will be striped when it is send to the dhcp namespace | |
| 23:20:58 | sean-k-mooney | so if you do a tcp dump on the bond you should see the request | |
| 23:21:06 | sean-k-mooney | if its gettign that far | |
| 23:22:11 | dking_desktop | Yes, I see them on the requests. In order to get ironic dnsmasq to work, though, I had to bring up the br-ex interface with an IP address. Could that be messing with this? | |
| 23:23:27 | sean-k-mooney | perhaps the br-ex normally should not require an ip | |
| 23:23:28 | dking_desktop | So, the baremetal server sends a DHCP request, it goes through the chassis switch, to the ToR, and then from there to the controller, and I see the data coming in on bond0. | |
| 23:24:07 | sean-k-mooney | so you deploed a second dnsmas for ironic | |
| 23:24:27 | sean-k-mooney | that is vaild but you have to set the dhcp provider i belive | |
| 23:24:34 | dking_desktop | Maybe not, but without it, I couldn't get ironic's dnsmasq to be able to see the packets. So, it was a hack. Would there have been a better way? Folks in the other channel were recommending that I have untagged packets tagged at the switch port, but so far, that's not been working. | |
| 23:25:04 | sean-k-mooney | so that is the old way to do it im not sure if its still required or the default. | |
| 23:25:26 | sean-k-mooney | when ironic was first created it handeld amost all its nteworking itself | |
| 23:25:32 | sean-k-mooney | then neutron was added after | |
| 23:25:47 | dking_desktop | Ironic handles its own dnsmasq. It works fine once I manually changed the interface to br-ex and put an IP on br-ex to bring it up. | |
| 23:25:50 | sean-k-mooney | slowly over the laft few years they have been moving ot useing neutorn where possible | |
| 23:26:01 | sean-k-mooney | ya | |
| 23:26:11 | sean-k-mooney | that was how i deployed previously | |
| 23:26:20 | sean-k-mooney | if you do a tcp dump on br-ex | |
| 23:26:25 | sean-k-mooney | do you see the dhcp request | |
| 23:27:01 | dking_desktop | Yes, I can see them on br-ex | |
| 23:27:24 | sean-k-mooney | and they are not vlan tagged | |
| 23:28:01 | dking_desktop | Correct | |
| 23:28:08 | sean-k-mooney | i have had issue with default route and arp that cause the respoces to not be sent by the br-ex in the past | |