| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-08 | |||
| 13:09:00 | sean-k-mooney | auto pep8 impleted as part of the E301 fixing | |
| 13:09:04 | lyarwood | sorry I thought you said it did, I see what you're sayingnow | |
| 13:13:14 | sean-k-mooney | lyarwood: summerised it in the review https://review.opendev.org/c/openstack/nova/+/806182/3#message-f37506ed0411c71d9ce66de251196048da171b34 | |
| 13:15:31 | gibi | sean-k-mooney: I have one question in autopep8 https://review.opendev.org/c/openstack/nova/+/806182/comment/e15200f9_a50f0219/ | |
| 13:16:08 | sean-k-mooney | we can add it to test-requiremetns yes | |
| 13:16:21 | sean-k-mooney | i just did not to avoid installing it in enves that did not need it | |
| 13:16:36 | sean-k-mooney | but if you prefer that i can update it shortly | |
| 13:16:41 | gibi | hm, will we ever need to pin the version of autopep8? | |
| 13:17:07 | sean-k-mooney | this can be done with upperconstraitns | |
| 13:17:11 | sean-k-mooney | or inline | |
| 13:17:36 | sean-k-mooney | i dont think so but its a vaild question | |
| 13:17:50 | gibi | currently it can only be inline afaik. but anyhow we don't pin it now, and we can move it to test-requirements when we need to pin it | |
| 13:17:59 | gibi | so I'm +2 | |
| 13:19:04 | sean-k-mooney | for line 69 yes since that does not include the base test env dep for the pep8 env the upper constratis form the base env still apply | |
| 13:22:56 | lyarwood | okay LGTM | |
| 13:23:27 | lyarwood | oh wait, downstream meeting :| | |
| 13:23:41 | lyarwood | helps if I look at the correct day in my cal | |
| 13:26:13 | opendevreview | Merged openstack/nova master: db: Remove legacy placement models https://review.opendev.org/c/openstack/nova/+/812146 | |
| 13:26:23 | opendevreview | Merged openstack/nova master: objects: Stop querying the main DB for keypairs https://review.opendev.org/c/openstack/nova/+/812147 | |
| 13:50:21 | bauzas | gibi: +w'd your series | |
| 13:50:27 | gibi | bauzas: thanks | |
| 14:13:47 | gibi | melwitt: hi! When you are up, we are waiting for you +A in https://review.opendev.org/c/openstack/nova/+/815689/2 so if you have time please check | |
| 14:23:57 | opendevreview | Merged openstack/nova master: Revert "tox: Encode specific Python versions" https://review.opendev.org/c/openstack/nova/+/804168 | |
| 14:30:31 | EugenMayer | When doing a nova backup, it seems like ephemeral disks are entirely excluded. is this to be expected? | |
| 14:30:51 | sean-k-mooney | EugenMayer: yes | |
| 14:31:06 | sean-k-mooney | EugenMayer: snapshots and backups are only done of the root disk | |
| 14:31:17 | EugenMayer | wow, that is a huge bummer ;/ | |
| 14:31:51 | EugenMayer | Is there any way to workarround that? somehow using glance directly and creating an image per disk? | |
| 14:31:52 | sean-k-mooney | ephemeral is not intended to be used for storing data that cant be recreated or safely lost | |
| 14:32:12 | sean-k-mooney | no it was exluded by design and cant be worked around | |
| 14:32:18 | sean-k-mooney | without code changes | |
| 14:32:28 | EugenMayer | Well, the problem is, there is no way to have multi-disk environment without defining one as ephemeral, right? | |
| 14:32:43 | sean-k-mooney | not without cinder | |
| 14:33:01 | EugenMayer | yes, those are volumes then | |
| 14:33:31 | sean-k-mooney | with openstack and most cloud plathform the expected and recommended usecase is to have only 1 disk with any addtional storage provided via volumes | |
| 14:34:20 | kevko | Hi nova team :) , testing wallaby and when I'm creating heat stack ..from time to time I see this error in nova .. is it known bug ? | |
| 14:34:21 | kevko | https://paste.opendev.org/show/810848/ | |
| 14:34:51 | sean-k-mooney | not in nova that normally means you have an issue with neutron | |
| 14:34:56 | EugenMayer | sean-k-mooney i understand that since one scale that one disk easily, no need to add others. But those VMs are legacy and partially it is part of the application desing to have 2 disks (on premise things). But well, still that now is somewhat a huge showstopper is was not accounting for yet | |
| 14:35:12 | dansmith | kevko: yeah, means neutron failed to send the even to nova to indicate that networking is ready | |
| 14:35:20 | sean-k-mooney | kevko: if you create really large heat stacks and your neutron installation is slow then it can time out wireing up the ports | |
| 14:36:01 | sean-k-mooney | you can workaround it by extending the timeout but the real fix is to tune your neutron deployment to support the workload | |
| 14:37:00 | kevko | well, it's really small heat template | |
| 14:37:44 | sean-k-mooney | is this in the upstream ci or a production issue in your deployment by the way | |
| 14:38:01 | sean-k-mooney | ah you said you were testing it so i guess its in your lab | |
| 14:38:10 | sean-k-mooney | what network backend are you using? | |
| 14:38:11 | EugenMayer | sean-k-mooney thank you for givin the definit answer. Not sure what i make out of it in terms of options/solutions but at least good to know it is intended and road blocked | |
| 14:38:24 | kevko | well, it is production problem also on client's side .. | |
| 14:38:31 | kevko | so i'm trying to replicate issue on lab | |
| 14:38:42 | kevko | i can't see relevant errors in neutron | |
| 14:39:15 | sean-k-mooney | kevko: ack is this ml2/ovn? ml2/ovs? cisco aci perhaps? | |
| 14:39:59 | sean-k-mooney | kevko: on the neutron side you likely wont see an error more it wont have sent the event before the timeout exired | |
| 14:40:37 | sean-k-mooney | kevko: if its close but not quite longenough you might actully see the network-vif-plugged event in the nova log after the timeout exired as an unexpected event | |
| 14:41:03 | dansmith | sean-k-mooney: we should debug log the time we waited each time so that people can see if they're close to the deadline normally, or if it's normally 5 seconds, one 300 timeout is obviously an event that's never coming | |
| 14:41:09 | kevko | ml2/ovs | |
| 14:41:17 | sean-k-mooney | if the env is using ml2/ovs you would need to check the neutron l2 agent log to see if it processed the port. for the other two you would hae to check the neutron server log i belive | |
| 14:43:54 | sean-k-mooney | dansmith: ya that is not a bad idea, on a related note i was speaking to bogdando downstream about also potentially queueing the event in the compute agent if no handeler with a TTL that might help in case that the event come before we start waiting but there are downsides to that. | |
| 14:44:29 | dansmith | sean-k-mooney: the entire point of that infrastructure is to make it hard to allow such things :) | |
| 14:44:47 | dansmith | sean-k-mooney: we prepare the event before we do the thing that could cause it to be fired | |
| 14:44:58 | sean-k-mooney | yep i know i mentioned that we did not do that for a reason and we discused it before and rejected it | |
| 14:44:58 | dansmith | otherwise we turn it into a statistical race condition problem | |
| 14:45:40 | sean-k-mooney | am by the way i would like your in put on the healthcheck design at some point | |
| 14:45:55 | dansmith | is something up to review? | |
| 14:46:11 | kevko | sean-k-mooney: regarding above ... timeout for rpc response ? or which timeout ..sorry :/ | |
| 14:46:55 | sean-k-mooney | not yet ill likely start on the spec tomorrow or later today but the open question i have is should the datastucture be storge in a module gobal or passed via the context to each fucntion | |
| 14:47:12 | sean-k-mooney | kevko: ill get you the link to the docs but there is a vif_plug_timeout you should increase | |
| 14:47:26 | sean-k-mooney | kevko: that is really jsut a workaround however | |
| 14:47:49 | kevko | so, bump number of neutron workers ? | |
| 14:47:54 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.vif_plugging_timeout | |
| 14:48:05 | dansmith | kevko: in all likelihood, you're not just a couple seconds late for a *5* minute timeout.. much more likely that it's never coming and changing the timeout will just make things more painful | |
| 14:48:19 | sean-k-mooney | yep | |
| 14:48:32 | sean-k-mooney | kevko: what you really need to do is look at the neutron agent logs on the compute | |
| 14:48:43 | dansmith | kevko: you should see messages in the nova-compute log saying that the event actually showed up but was discarded - if you don't, then changing the timeout just makes your system worse | |
| 14:48:46 | sean-k-mooney | and see when the port was added and how far it got in wiring it up | |
| 14:52:59 | kevko | sean-k-mooney: same log + neutron-ovs-agent 10 minutes +/- ago https://paste.opendev.org/show/810850/ | |
| 14:54:04 | sean-k-mooney | kevko: are you using iptbals by the way or openvswtich firewall driver | |
| 14:54:13 | kevko | ovs | |
| 14:54:36 | sean-k-mooney | ok in that case in wallaby the port is plugged by libvirt as part of the instance creation | |
| 14:54:40 | sean-k-mooney | gibi: o/ | |
| 14:55:03 | opendevreview | Dan Smith proposed openstack/nova master: Log VIF event wait times https://review.opendev.org/c/openstack/nova/+/817030 | |
| 14:55:10 | dansmith | sean-k-mooney: ^ | |
| 14:55:15 | sean-k-mooney | kevko: at least presently since i have not backported the change to delegate taht to os-vif yet | |
| 14:56:05 | sean-k-mooney | oh you are using the Stopwatch functionality instead of time.now and subtracting ya that is nicer | |
| 14:56:36 | kevko | sean-k-mooney: what does it mean ? :/ | |
| 14:56:45 | kevko | checking bad logs ? | |
| 14:57:10 | sean-k-mooney | kevko: that was to dansmith he has already created a patch to log the elapsed time | |
| 14:57:21 | sean-k-mooney | kevko: https://review.opendev.org/c/openstack/nova/+/817030/1/nova/compute/manager.py#491 | |
| 14:57:45 | sean-k-mooney | kevko: oh you ment about libvirt plugging the interafce not os-vif | |
| 14:57:58 | kevko | yeah | |
| 14:58:54 | sean-k-mooney | in wallaby when we call plug on os-vif for port wiht hybrid-plug=false, such as when using the ovs firewall, os-vif just ensure the ovs bridge exists and the port is actully added to the bridge by libvirt | |
| 14:59:15 | sean-k-mooney | so the port wont be added until libvirt tries to start the vm | |
| 15:00:01 | sean-k-mooney | the neutron agent only starts wiring up the port after its created by libvirt but in this code path it will happen only after libvirt starts the vm | |
| 15:00:15 | sean-k-mooney | in the pasued state | |
| 15:00:58 | sean-k-mooney | if you were usign a differnt network backedn the behaivor would be differnt in nova | |
| 15:01:13 | sean-k-mooney | well potentially in any case which is why i asked | |
| 15:03:37 | sean-k-mooney | with your configuration the port will only be created at this point https://github.com/openstack/nova/blob/400d25fdeb45fe53be1069996ffaa3783eb4402b/nova/virt/libvirt/driver.py#L7209-L7212 | |
| 15:03:53 | sean-k-mooney | for other configurtion it would happen eairler here https://github.com/openstack/nova/blob/400d25fdeb45fe53be1069996ffaa3783eb4402b/nova/virt/libvirt/driver.py#L7205 | |
| 15:05:13 | sean-k-mooney | kevko: if creating the guest took a long time its possible that we woudl time out waiting for the even but that is a low proablity | |
| 15:06:38 | sean-k-mooney | kevko: if you put the neutron l2 agent in debug mode it will print the addtion and removal of the port in the ovsdb and also message as it configures them | |
| 15:06:51 | sean-k-mooney | *debug log level | |