Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-13
23:12:40 sean-k-mooney and the tftp server
23:13:47 sean-k-mooney once the ironic node is provisioned it will normally use a different interface for the teant to ssh in/have netwrok conenctivity out onto the datacenter
23:15:31 sean-k-mooney dking_desktop: you might be hitting this by the way https://docs.openstack.org/ironic/train/admin/troubleshooting.html#dhcp-during-pxe-or-ipxe-is-inconsistent-or-unreliable
23:15:46 dking_desktop So, maybe you can help me here. Inside of the network namespace, I'm not seeing any DHCP requests. That explains why I didn't see anything logged and no responses.
23:16:41 sean-k-mooney ya so its possible the dhcp request is being droped by the switch before it gets to the contoler
23:17:23 dking_desktop So, how _should_ the packets be getting there? I see that this network interface is an ovs port inside of br-int. I know that br-int is patched to br-ex.
23:18:22 sean-k-mooney yes and the br-ex should have a physical interface attached
23:18:23 dking_desktop The server is booting up using DHCP/PXE, but it is on a trunked port, so the packets are coming in untagged. I know that's caused me trouble before.
23:18:49 sean-k-mooney right so if the neutron network is a flat netwrok
23:19:14 sean-k-mooney then it should be untag form the server, get to the top of rack switch and remain untagged
23:19:23 sean-k-mooney then as ita a broadcast it will flood
23:19:31 dking_desktop It does. It's attached to bond0. So, does br-ex send DHCP broadcasts to br-int, and then it sends them to all of its ports? That doesn't sound right.
23:19:31 sean-k-mooney eventually make it to the contoler
23:20:23 sean-k-mooney when it arrives in the contoler it will enter the br-ex. it will be vlan taged with a local vlan and then be flooded to only the ports fo that vlan
23:20:39 sean-k-mooney then it will be striped when it is send to the dhcp namespace
23:20:58 sean-k-mooney so if you do a tcp dump on the bond you should see the request
23:21:06 sean-k-mooney if its gettign that far
23:22:11 dking_desktop Yes, I see them on the requests. In order to get ironic dnsmasq to work, though, I had to bring up the br-ex interface with an IP address. Could that be messing with this?
23:23:27 sean-k-mooney perhaps the br-ex normally should not require an ip
23:23:28 dking_desktop So, the baremetal server sends a DHCP request, it goes through the chassis switch, to the ToR, and then from there to the controller, and I see the data coming in on bond0.
23:24:07 sean-k-mooney so you deploed a second dnsmas for ironic
23:24:27 sean-k-mooney that is vaild but you have to set the dhcp provider i belive
23:24:34 dking_desktop Maybe not, but without it, I couldn't get ironic's dnsmasq to be able to see the packets. So, it was a hack. Would there have been a better way? Folks in the other channel were recommending that I have untagged packets tagged at the switch port, but so far, that's not been working.
23:25:04 sean-k-mooney so that is the old way to do it im not sure if its still required or the default.
23:25:26 sean-k-mooney when ironic was first created it handeld amost all its nteworking itself
23:25:32 sean-k-mooney then neutron was added after
23:25:47 dking_desktop Ironic handles its own dnsmasq. It works fine once I manually changed the interface to br-ex and put an IP on br-ex to bring it up.
23:25:50 sean-k-mooney slowly over the laft few years they have been moving ot useing neutorn where possible
23:26:01 sean-k-mooney ya
23:26:11 sean-k-mooney that was how i deployed previously
23:26:20 sean-k-mooney if you do a tcp dump on br-ex
23:26:25 sean-k-mooney do you see the dhcp request
23:27:01 dking_desktop Yes, I can see them on br-ex
23:27:24 sean-k-mooney and they are not vlan tagged
23:28:01 dking_desktop Correct
23:28:08 sean-k-mooney i have had issue with default route and arp that cause the respoces to not be sent by the br-ex in the past
23:28:16 sean-k-mooney do you have a second interface on the same subnet
23:28:56 dking_desktop Which subnet is that? The one that I setup for br-ex?
23:29:04 sean-k-mooney yes
23:29:54 sean-k-mooney i have had issues in the past where i have added an ip to the br-ex and recived packet but had teh reply sent via ens3 becaue it had an ip in the same subnet but a better metric
23:29:54 dking_desktop I don't think so. I set that up to use an IP from the range I've been using for untagged packets.
23:30:57 dking_desktop No, the only route for that subnet is through dev br-ex.
23:31:19 sean-k-mooney ok this sounds very familar but i dont recall the casue.
23:31:44 dking_desktop From inside of the qdhcp-* subnet, "tcpdump -i any -nne -xx -Avvvv" hasn't shown any packets yet.
23:32:30 sean-k-mooney yes so i dont think it will since you the iniall boot will go to the provisoing network
23:32:48 sean-k-mooney i suspect if you check the uuid that is the dhcp agent for the tenat network
23:32:53 sean-k-mooney not the provisoning network
23:33:38 sean-k-mooney anyway its getting late and im out of ideas so ill have to leave it there
23:35:14 dking_desktop Oh, okay. I see now that the * in qdhcp-* is actually an ID for a network. Exactly, it's the tenant network.
23:35:22 dking_desktop Let me check that I setup DHCP for the provisioning network.
23:36:36 dking_desktop Oh, I did not enable DHCP for the provisioning network. I can enable it. Is that what's supposed to happen?
23:36:39 sean-k-mooney if you didnt enable it in the subnet then neutron would not create the namespce or spwan the dnsmas process for it so that could be it
23:37:27 sean-k-mooney there is one way to find out:) but i think if you use the nutron netorking interface dirver then yes you shoudl turn it on
23:37:50 dking_desktop Let me try that. But if that's the case, would the DNS be the right one, with the PXE information in it?
23:37:53 sean-k-mooney if you use the flat network interface dirver i think you deploy a seperate dnsmask for ironic as you did manually
23:38:42 sean-k-mooney honestly im at the edge of my knoladge here as i said its been a while but i cant recall
23:42:19 dking_desktop Your help has been very enlightening. Also, I just saw some packets in that network, and it does seem to be set up for PXE. I'm going to try creating a server and see if that works.
23:42:34 dking_desktop But even if not, I've learned much, so thank you very much!
23:45:04 sean-k-mooney this is realy really old but if you have not seen it before its how neutron ovs networking used to work
23:45:06 sean-k-mooney https://www.rdoproject.org/networking/networking-in-too-much-detail/
23:45:17 sean-k-mooney its now simpler
23:45:43 sean-k-mooney but it is a good thing to read over at least once even if its not how it works exactly today
23:48:18 dking_desktop Thank you. I'll check that out. It may help with some of the dark spots in my knowledge.
#openstack-nova - 2020-02-14
00:59:21 huaqiang stephenfin: thank you and all for the review and patient.
01:11:25 huaqiang s/patient/patience/ :)
01:28:18 brinzhang efried: I think https://review.opendev.org/#/c/580336 all things are ready in SPEC, and also provide it's PoC codes https://review.opendev.org/#/c/693828/ agreement reached at Shanghai PTG https://etherpad.openstack.org/p/nova-shanghai-ptg(Line253)
01:31:29 johnsom Hi nova folks. FYI, I opened a bug that anti-affinity appears to have been broken since at least August. https://bugs.launchpad.net/nova/+bug/1863190 Let me know if there is more information I should collect while I have the instance up and running.
01:31:29 openstack Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New]
02:09:05 melwitt johnsom: did you boot the two instances at the same time or near the same time? did you use multi-create?
02:10:06 johnsom They were close in time, but individual calls to the nova api
02:10:32 johnsom Our normal active/standby boot sequence
02:11:21 melwitt ok. I'm wondering if it's a race basically
02:11:55 melwitt like if you wait for the first instance to become active and then do another request, would it fail
02:11:58 johnsom Yeah, probably. This used to work.
02:12:41 johnsom Yeah, I don’t know about waiting for active. That can take some time.
02:13:46 johnsom We have seen that up to five minutes, mostly scheduler time. I don’t think we want to hold up the boot process for the secondary that long
02:14:30 melwitt no sorry, I meant in your devstack
02:15:15 melwitt I can try it later too but not tonight. I was just asking in case you had that easy environment already up
02:15:21 johnsom Ah, as a test. I can try that. It will be tomorrow however, it is dinner time. Lol
02:15:27 melwitt same
02:15:52 johnsom If there is a list of thing you would like me to try, comment on the bug and I will run them tomorrow
02:15:57 melwitt there's a thing called the "late affinity check" in nova-compute that is on by default that should handle races
02:16:35 melwitt I don't know off the top of my head what could have regressed this.
02:18:03 melwitt I'll try out some things with a devstack tomorrow to start finding out what's going on
02:18:46 melwitt efried: potential regression alert ^ fyi https://bugs.launchpad.net/nova/+bug/1863190
02:18:46 openstack Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New]
02:18:53 johnsom Ok, thanks. I will hold this instance through tomorrow in case we need it
02:21:14 melwitt k thanks
02:55:29 openstackgerrit Liang Fang proposed openstack/nova-specs master: Support volume local cache https://review.opendev.org/689070
02:58:01 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751
06:41:43 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751
06:42:13 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751
07:57:56 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777
07:58:47 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751
08:59:44 openstackgerrit Guo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120
09:03:10 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777
09:07:21 openstackgerrit Guo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120
09:28:53 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751
10:02:02 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM - Test TEMPEST_EXTEND_ATTACHED_ENCRYPTED_VOLUME https://review.opendev.org/707593
10:11:38 openstackgerrit Merged openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226

Earlier   Later