| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-08 | |||
| 14:34:20 | kevko | Hi nova team :) , testing wallaby and when I'm creating heat stack ..from time to time I see this error in nova .. is it known bug ? | |
| 14:34:21 | kevko | https://paste.opendev.org/show/810848/ | |
| 14:34:51 | sean-k-mooney | not in nova that normally means you have an issue with neutron | |
| 14:34:56 | EugenMayer | sean-k-mooney i understand that since one scale that one disk easily, no need to add others. But those VMs are legacy and partially it is part of the application desing to have 2 disks (on premise things). But well, still that now is somewhat a huge showstopper is was not accounting for yet | |
| 14:35:12 | dansmith | kevko: yeah, means neutron failed to send the even to nova to indicate that networking is ready | |
| 14:35:20 | sean-k-mooney | kevko: if you create really large heat stacks and your neutron installation is slow then it can time out wireing up the ports | |
| 14:36:01 | sean-k-mooney | you can workaround it by extending the timeout but the real fix is to tune your neutron deployment to support the workload | |
| 14:37:00 | kevko | well, it's really small heat template | |
| 14:37:44 | sean-k-mooney | is this in the upstream ci or a production issue in your deployment by the way | |
| 14:38:01 | sean-k-mooney | ah you said you were testing it so i guess its in your lab | |
| 14:38:10 | sean-k-mooney | what network backend are you using? | |
| 14:38:11 | EugenMayer | sean-k-mooney thank you for givin the definit answer. Not sure what i make out of it in terms of options/solutions but at least good to know it is intended and road blocked | |
| 14:38:24 | kevko | well, it is production problem also on client's side .. | |
| 14:38:31 | kevko | so i'm trying to replicate issue on lab | |
| 14:38:42 | kevko | i can't see relevant errors in neutron | |
| 14:39:15 | sean-k-mooney | kevko: ack is this ml2/ovn? ml2/ovs? cisco aci perhaps? | |
| 14:39:59 | sean-k-mooney | kevko: on the neutron side you likely wont see an error more it wont have sent the event before the timeout exired | |
| 14:40:37 | sean-k-mooney | kevko: if its close but not quite longenough you might actully see the network-vif-plugged event in the nova log after the timeout exired as an unexpected event | |
| 14:41:03 | dansmith | sean-k-mooney: we should debug log the time we waited each time so that people can see if they're close to the deadline normally, or if it's normally 5 seconds, one 300 timeout is obviously an event that's never coming | |
| 14:41:09 | kevko | ml2/ovs | |
| 14:41:17 | sean-k-mooney | if the env is using ml2/ovs you would need to check the neutron l2 agent log to see if it processed the port. for the other two you would hae to check the neutron server log i belive | |
| 14:43:54 | sean-k-mooney | dansmith: ya that is not a bad idea, on a related note i was speaking to bogdando downstream about also potentially queueing the event in the compute agent if no handeler with a TTL that might help in case that the event come before we start waiting but there are downsides to that. | |
| 14:44:29 | dansmith | sean-k-mooney: the entire point of that infrastructure is to make it hard to allow such things :) | |
| 14:44:47 | dansmith | sean-k-mooney: we prepare the event before we do the thing that could cause it to be fired | |
| 14:44:58 | dansmith | otherwise we turn it into a statistical race condition problem | |
| 14:44:58 | sean-k-mooney | yep i know i mentioned that we did not do that for a reason and we discused it before and rejected it | |
| 14:45:40 | sean-k-mooney | am by the way i would like your in put on the healthcheck design at some point | |
| 14:45:55 | dansmith | is something up to review? | |
| 14:46:11 | kevko | sean-k-mooney: regarding above ... timeout for rpc response ? or which timeout ..sorry :/ | |
| 14:46:55 | sean-k-mooney | not yet ill likely start on the spec tomorrow or later today but the open question i have is should the datastucture be storge in a module gobal or passed via the context to each fucntion | |
| 14:47:12 | sean-k-mooney | kevko: ill get you the link to the docs but there is a vif_plug_timeout you should increase | |
| 14:47:26 | sean-k-mooney | kevko: that is really jsut a workaround however | |
| 14:47:49 | kevko | so, bump number of neutron workers ? | |
| 14:47:54 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.vif_plugging_timeout | |
| 14:48:05 | dansmith | kevko: in all likelihood, you're not just a couple seconds late for a *5* minute timeout.. much more likely that it's never coming and changing the timeout will just make things more painful | |
| 14:48:19 | sean-k-mooney | yep | |
| 14:48:32 | sean-k-mooney | kevko: what you really need to do is look at the neutron agent logs on the compute | |
| 14:48:43 | dansmith | kevko: you should see messages in the nova-compute log saying that the event actually showed up but was discarded - if you don't, then changing the timeout just makes your system worse | |
| 14:48:46 | sean-k-mooney | and see when the port was added and how far it got in wiring it up | |
| 14:52:59 | kevko | sean-k-mooney: same log + neutron-ovs-agent 10 minutes +/- ago https://paste.opendev.org/show/810850/ | |
| 14:54:04 | sean-k-mooney | kevko: are you using iptbals by the way or openvswtich firewall driver | |
| 14:54:13 | kevko | ovs | |
| 14:54:36 | sean-k-mooney | ok in that case in wallaby the port is plugged by libvirt as part of the instance creation | |
| 14:54:40 | sean-k-mooney | gibi: o/ | |
| 14:55:03 | opendevreview | Dan Smith proposed openstack/nova master: Log VIF event wait times https://review.opendev.org/c/openstack/nova/+/817030 | |
| 14:55:10 | dansmith | sean-k-mooney: ^ | |
| 14:55:15 | sean-k-mooney | kevko: at least presently since i have not backported the change to delegate taht to os-vif yet | |
| 14:56:05 | sean-k-mooney | oh you are using the Stopwatch functionality instead of time.now and subtracting ya that is nicer | |
| 14:56:36 | kevko | sean-k-mooney: what does it mean ? :/ | |
| 14:56:45 | kevko | checking bad logs ? | |
| 14:57:10 | sean-k-mooney | kevko: that was to dansmith he has already created a patch to log the elapsed time | |
| 14:57:21 | sean-k-mooney | kevko: https://review.opendev.org/c/openstack/nova/+/817030/1/nova/compute/manager.py#491 | |
| 14:57:45 | sean-k-mooney | kevko: oh you ment about libvirt plugging the interafce not os-vif | |
| 14:57:58 | kevko | yeah | |
| 14:58:54 | sean-k-mooney | in wallaby when we call plug on os-vif for port wiht hybrid-plug=false, such as when using the ovs firewall, os-vif just ensure the ovs bridge exists and the port is actully added to the bridge by libvirt | |
| 14:59:15 | sean-k-mooney | so the port wont be added until libvirt tries to start the vm | |
| 15:00:01 | sean-k-mooney | the neutron agent only starts wiring up the port after its created by libvirt but in this code path it will happen only after libvirt starts the vm | |
| 15:00:15 | sean-k-mooney | in the pasued state | |
| 15:00:58 | sean-k-mooney | if you were usign a differnt network backedn the behaivor would be differnt in nova | |
| 15:01:13 | sean-k-mooney | well potentially in any case which is why i asked | |
| 15:03:37 | sean-k-mooney | with your configuration the port will only be created at this point https://github.com/openstack/nova/blob/400d25fdeb45fe53be1069996ffaa3783eb4402b/nova/virt/libvirt/driver.py#L7209-L7212 | |
| 15:03:53 | sean-k-mooney | for other configurtion it would happen eairler here https://github.com/openstack/nova/blob/400d25fdeb45fe53be1069996ffaa3783eb4402b/nova/virt/libvirt/driver.py#L7205 | |
| 15:05:13 | sean-k-mooney | kevko: if creating the guest took a long time its possible that we woudl time out waiting for the even but that is a low proablity | |
| 15:06:38 | sean-k-mooney | kevko: if you put the neutron l2 agent in debug mode it will print the addtion and removal of the port in the ovsdb and also message as it configures them | |
| 15:06:51 | sean-k-mooney | *debug log level | |
| 15:06:54 | kevko | hmm, so issue is not in workers .. | |
| 15:07:08 | sean-k-mooney | kevko: likely not | |
| 15:07:48 | kevko | ok, i will try to turn on neutron debug log and see | |
| 15:07:58 | sean-k-mooney | kevko: i would suggest putting noa and neutron into debug mode on the host then booting another vm and check if while its waitign the domain is succesfully created in libvirt and the tap device is added to ovs | |
| 15:08:22 | sean-k-mooney | if it is then you shoudl check the agent log to see if it deteched it and started to process it | |
| 15:16:54 | kevko | thank you sean-k-mooney, probably i will ask you tomorrow | |
| 15:17:27 | sean-k-mooney | lyarwood: by the way im just stack a clean devstack then im going to work on the qemu wapper that we discussed last week. ill ping you when i have something | |
| 15:21:19 | opendevreview | Elod Illes proposed openstack/nova stable/stein: Reject open redirection in the console proxy https://review.opendev.org/c/openstack/nova/+/802935 | |
| 15:29:01 | lyarwood | sean-k-mooney: ack sounds good | |
| 15:29:19 | opendevreview | Elod Illes proposed openstack/nova stable/stein: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/817037 | |
| 15:56:37 | dansmith | sean-k-mooney: gmann: So, going back to last week's discussion about extra specs... | |
| 15:57:11 | dansmith | sounds like system reader and project admin should be able to see extra specs at this point to be the most compatible with existing stuff yeah? | |
| 15:57:39 | dansmith | (i.e. any system user can see them, only admin on the project side can see them) | |
| 15:58:51 | sean-k-mooney | i think project reader shoudl be able to see extra specs | |
| 15:59:04 | sean-k-mooney | i dont think you should need project admin | |
| 15:59:15 | dansmith | I know you do, but today regular users can't right? | |
| 15:59:22 | sean-k-mooney | they can | |
| 15:59:40 | sean-k-mooney | its systrem_reader_or_project_reader i think today | |
| 15:59:56 | dansmith | oh, maybe that explains why the test is so weird | |
| 16:00:07 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/policies/flavor_extra_specs.py#L78 | |
| 16:00:29 | dansmith | heh yeah okay | |
| 16:00:46 | sean-k-mooney | adding/removing obviously should be system scoped i think | |
| 16:01:06 | dansmith | the test conflates a lot of stuff, I guess for that reason, so I'll have to split things apart a bit because system can't see the servers, and thus can't see the embedded flavor | |
| 16:01:29 | sean-k-mooney | ah right | |
| 16:12:25 | dansmith | hrm, | |
| 16:12:43 | dansmith | pretty sure there's a bug in the create test that is testing index perms for create | |
| 16:13:32 | dansmith | https://github.com/openstack/nova/blob/171138146a648d22474b7021ac730e26f03455f8/nova/tests/unit/policies/test_flavor_extra_specs.py#L235-L254 | |
| 16:13:43 | dansmith | the test purports to check update, | |
| 16:13:52 | melwitt | gibi: ack, will look | |
| 16:13:58 | dansmith | but it's actually checking index, which is rule_name instead of rule | |
| 16:14:13 | dansmith | and it's stubbing out update for everybody instead of index | |
| 16:14:32 | opendevreview | Merged openstack/nova stable/victoria: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810914 | |
| 16:15:16 | dansmith | and the test actually asserts that system reader can update extra specs | |
| 16:18:28 | lyarwood | elodilles / gibi ; https://review.opendev.org/q/I26b2a14e0b91c0ab77299c3e4fbed5f7916fe8cf do either of you recall why we don't need this on >= stable/victoria ? | |
| 16:21:26 | lyarwood | seeing some weird behaviour downstream where we hit the 2to3 issues on py39 thanks to virtualenv and setuptools versions but upstream it looks like UC is downgrading setuptools for us during the run | |