| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-13 | |||
| 23:10:30 | dking_desktop | I might need to do that. Right now, I want to leave my options open. | |
| 23:11:37 | dking_desktop | Is the "provisioning_network" only to get the ramdisk booted and deploy the server? So, once that's done, it's either not necessary, or perhaps only for status updates? | |
| 23:12:03 | sean-k-mooney | yes basicaly | |
| 23:12:28 | sean-k-mooney | it is the network that need to have conectivity to where the image is located | |
| 23:12:40 | sean-k-mooney | and the tftp server | |
| 23:13:47 | sean-k-mooney | once the ironic node is provisioned it will normally use a different interface for the teant to ssh in/have netwrok conenctivity out onto the datacenter | |
| 23:15:31 | sean-k-mooney | dking_desktop: you might be hitting this by the way https://docs.openstack.org/ironic/train/admin/troubleshooting.html#dhcp-during-pxe-or-ipxe-is-inconsistent-or-unreliable | |
| 23:15:46 | dking_desktop | So, maybe you can help me here. Inside of the network namespace, I'm not seeing any DHCP requests. That explains why I didn't see anything logged and no responses. | |
| 23:16:41 | sean-k-mooney | ya so its possible the dhcp request is being droped by the switch before it gets to the contoler | |
| 23:17:23 | dking_desktop | So, how _should_ the packets be getting there? I see that this network interface is an ovs port inside of br-int. I know that br-int is patched to br-ex. | |
| 23:18:22 | sean-k-mooney | yes and the br-ex should have a physical interface attached | |
| 23:18:23 | dking_desktop | The server is booting up using DHCP/PXE, but it is on a trunked port, so the packets are coming in untagged. I know that's caused me trouble before. | |
| 23:18:49 | sean-k-mooney | right so if the neutron network is a flat netwrok | |
| 23:19:14 | sean-k-mooney | then it should be untag form the server, get to the top of rack switch and remain untagged | |
| 23:19:23 | sean-k-mooney | then as ita a broadcast it will flood | |
| 23:19:31 | sean-k-mooney | eventually make it to the contoler | |
| 23:19:31 | dking_desktop | It does. It's attached to bond0. So, does br-ex send DHCP broadcasts to br-int, and then it sends them to all of its ports? That doesn't sound right. | |
| 23:20:23 | sean-k-mooney | when it arrives in the contoler it will enter the br-ex. it will be vlan taged with a local vlan and then be flooded to only the ports fo that vlan | |
| 23:20:39 | sean-k-mooney | then it will be striped when it is send to the dhcp namespace | |
| 23:20:58 | sean-k-mooney | so if you do a tcp dump on the bond you should see the request | |
| 23:21:06 | sean-k-mooney | if its gettign that far | |
| 23:22:11 | dking_desktop | Yes, I see them on the requests. In order to get ironic dnsmasq to work, though, I had to bring up the br-ex interface with an IP address. Could that be messing with this? | |
| 23:23:27 | sean-k-mooney | perhaps the br-ex normally should not require an ip | |
| 23:23:28 | dking_desktop | So, the baremetal server sends a DHCP request, it goes through the chassis switch, to the ToR, and then from there to the controller, and I see the data coming in on bond0. | |
| 23:24:07 | sean-k-mooney | so you deploed a second dnsmas for ironic | |
| 23:24:27 | sean-k-mooney | that is vaild but you have to set the dhcp provider i belive | |
| 23:24:34 | dking_desktop | Maybe not, but without it, I couldn't get ironic's dnsmasq to be able to see the packets. So, it was a hack. Would there have been a better way? Folks in the other channel were recommending that I have untagged packets tagged at the switch port, but so far, that's not been working. | |
| 23:25:04 | sean-k-mooney | so that is the old way to do it im not sure if its still required or the default. | |
| 23:25:26 | sean-k-mooney | when ironic was first created it handeld amost all its nteworking itself | |
| 23:25:32 | sean-k-mooney | then neutron was added after | |
| 23:25:47 | dking_desktop | Ironic handles its own dnsmasq. It works fine once I manually changed the interface to br-ex and put an IP on br-ex to bring it up. | |
| 23:25:50 | sean-k-mooney | slowly over the laft few years they have been moving ot useing neutorn where possible | |
| 23:26:01 | sean-k-mooney | ya | |
| 23:26:11 | sean-k-mooney | that was how i deployed previously | |
| 23:26:20 | sean-k-mooney | if you do a tcp dump on br-ex | |
| 23:26:25 | sean-k-mooney | do you see the dhcp request | |
| 23:27:01 | dking_desktop | Yes, I can see them on br-ex | |
| 23:27:24 | sean-k-mooney | and they are not vlan tagged | |
| 23:28:01 | dking_desktop | Correct | |
| 23:28:08 | sean-k-mooney | i have had issue with default route and arp that cause the respoces to not be sent by the br-ex in the past | |
| 23:28:16 | sean-k-mooney | do you have a second interface on the same subnet | |
| 23:28:56 | dking_desktop | Which subnet is that? The one that I setup for br-ex? | |
| 23:29:04 | sean-k-mooney | yes | |
| 23:29:54 | dking_desktop | I don't think so. I set that up to use an IP from the range I've been using for untagged packets. | |
| 23:29:54 | sean-k-mooney | i have had issues in the past where i have added an ip to the br-ex and recived packet but had teh reply sent via ens3 becaue it had an ip in the same subnet but a better metric | |
| 23:30:57 | dking_desktop | No, the only route for that subnet is through dev br-ex. | |
| 23:31:19 | sean-k-mooney | ok this sounds very familar but i dont recall the casue. | |
| 23:31:44 | dking_desktop | From inside of the qdhcp-* subnet, "tcpdump -i any -nne -xx -Avvvv" hasn't shown any packets yet. | |
| 23:32:30 | sean-k-mooney | yes so i dont think it will since you the iniall boot will go to the provisoing network | |
| 23:32:48 | sean-k-mooney | i suspect if you check the uuid that is the dhcp agent for the tenat network | |
| 23:32:53 | sean-k-mooney | not the provisoning network | |
| 23:33:38 | sean-k-mooney | anyway its getting late and im out of ideas so ill have to leave it there | |
| 23:35:14 | dking_desktop | Oh, okay. I see now that the * in qdhcp-* is actually an ID for a network. Exactly, it's the tenant network. | |
| 23:35:22 | dking_desktop | Let me check that I setup DHCP for the provisioning network. | |
| 23:36:36 | dking_desktop | Oh, I did not enable DHCP for the provisioning network. I can enable it. Is that what's supposed to happen? | |
| 23:36:39 | sean-k-mooney | if you didnt enable it in the subnet then neutron would not create the namespce or spwan the dnsmas process for it so that could be it | |
| 23:37:27 | sean-k-mooney | there is one way to find out:) but i think if you use the nutron netorking interface dirver then yes you shoudl turn it on | |
| 23:37:50 | dking_desktop | Let me try that. But if that's the case, would the DNS be the right one, with the PXE information in it? | |
| 23:37:53 | sean-k-mooney | if you use the flat network interface dirver i think you deploy a seperate dnsmask for ironic as you did manually | |
| 23:38:42 | sean-k-mooney | honestly im at the edge of my knoladge here as i said its been a while but i cant recall | |
| 23:42:19 | dking_desktop | Your help has been very enlightening. Also, I just saw some packets in that network, and it does seem to be set up for PXE. I'm going to try creating a server and see if that works. | |
| 23:42:34 | dking_desktop | But even if not, I've learned much, so thank you very much! | |
| 23:45:04 | sean-k-mooney | this is realy really old but if you have not seen it before its how neutron ovs networking used to work | |
| 23:45:06 | sean-k-mooney | https://www.rdoproject.org/networking/networking-in-too-much-detail/ | |
| 23:45:17 | sean-k-mooney | its now simpler | |
| 23:45:43 | sean-k-mooney | but it is a good thing to read over at least once even if its not how it works exactly today | |
| 23:48:18 | dking_desktop | Thank you. I'll check that out. It may help with some of the dark spots in my knowledge. | |
| #openstack-nova - 2020-02-14 | |||
| 00:59:21 | huaqiang | stephenfin: thank you and all for the review and patient. | |
| 01:11:25 | huaqiang | s/patient/patience/ :) | |
| 01:28:18 | brinzhang | efried: I think https://review.opendev.org/#/c/580336 all things are ready in SPEC, and also provide it's PoC codes https://review.opendev.org/#/c/693828/ agreement reached at Shanghai PTG https://etherpad.openstack.org/p/nova-shanghai-ptg(Line253) | |
| 01:31:29 | openstack | Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New] | |
| 01:31:29 | johnsom | Hi nova folks. FYI, I opened a bug that anti-affinity appears to have been broken since at least August. https://bugs.launchpad.net/nova/+bug/1863190 Let me know if there is more information I should collect while I have the instance up and running. | |
| 02:09:05 | melwitt | johnsom: did you boot the two instances at the same time or near the same time? did you use multi-create? | |
| 02:10:06 | johnsom | They were close in time, but individual calls to the nova api | |
| 02:10:32 | johnsom | Our normal active/standby boot sequence | |
| 02:11:21 | melwitt | ok. I'm wondering if it's a race basically | |
| 02:11:55 | melwitt | like if you wait for the first instance to become active and then do another request, would it fail | |
| 02:11:58 | johnsom | Yeah, probably. This used to work. | |
| 02:12:41 | johnsom | Yeah, I don’t know about waiting for active. That can take some time. | |
| 02:13:46 | johnsom | We have seen that up to five minutes, mostly scheduler time. I don’t think we want to hold up the boot process for the secondary that long | |
| 02:14:30 | melwitt | no sorry, I meant in your devstack | |
| 02:15:15 | melwitt | I can try it later too but not tonight. I was just asking in case you had that easy environment already up | |
| 02:15:21 | johnsom | Ah, as a test. I can try that. It will be tomorrow however, it is dinner time. Lol | |
| 02:15:27 | melwitt | same | |
| 02:15:52 | johnsom | If there is a list of thing you would like me to try, comment on the bug and I will run them tomorrow | |
| 02:15:57 | melwitt | there's a thing called the "late affinity check" in nova-compute that is on by default that should handle races | |
| 02:16:35 | melwitt | I don't know off the top of my head what could have regressed this. | |
| 02:18:03 | melwitt | I'll try out some things with a devstack tomorrow to start finding out what's going on | |
| 02:18:46 | openstack | Launchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New] | |
| 02:18:46 | melwitt | efried: potential regression alert ^ fyi https://bugs.launchpad.net/nova/+bug/1863190 | |
| 02:18:53 | johnsom | Ok, thanks. I will hold this instance through tomorrow in case we need it | |
| 02:21:14 | melwitt | k thanks | |
| 02:55:29 | openstackgerrit | Liang Fang proposed openstack/nova-specs master: Support volume local cache https://review.opendev.org/689070 | |
| 02:58:01 | openstackgerrit | Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751 | |
| 06:41:43 | openstackgerrit | Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751 | |
| 06:42:13 | openstackgerrit | Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751 | |
| 07:57:56 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |
| 07:58:47 | openstackgerrit | Brin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy https://review.opendev.org/707751 | |
| 08:59:44 | openstackgerrit | Guo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120 | |
| 09:03:10 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |