Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-08
14:45:40 sean-k-mooney am by the way i would like your in put on the healthcheck design at some point
14:45:55 dansmith is something up to review?
14:46:11 kevko sean-k-mooney: regarding above ... timeout for rpc response ? or which timeout ..sorry :/
14:46:55 sean-k-mooney not yet ill likely start on the spec tomorrow or later today but the open question i have is should the datastucture be storge in a module gobal or passed via the context to each fucntion
14:47:12 sean-k-mooney kevko: ill get you the link to the docs but there is a vif_plug_timeout you should increase
14:47:26 sean-k-mooney kevko: that is really jsut a workaround however
14:47:49 kevko so, bump number of neutron workers ?
14:47:54 sean-k-mooney https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.vif_plugging_timeout
14:48:05 dansmith kevko: in all likelihood, you're not just a couple seconds late for a *5* minute timeout.. much more likely that it's never coming and changing the timeout will just make things more painful
14:48:19 sean-k-mooney yep
14:48:32 sean-k-mooney kevko: what you really need to do is look at the neutron agent logs on the compute
14:48:43 dansmith kevko: you should see messages in the nova-compute log saying that the event actually showed up but was discarded - if you don't, then changing the timeout just makes your system worse
14:48:46 sean-k-mooney and see when the port was added and how far it got in wiring it up
14:52:59 kevko sean-k-mooney: same log + neutron-ovs-agent 10 minutes +/- ago https://paste.opendev.org/show/810850/
14:54:04 sean-k-mooney kevko: are you using iptbals by the way or openvswtich firewall driver
14:54:13 kevko ovs
14:54:36 sean-k-mooney ok in that case in wallaby the port is plugged by libvirt as part of the instance creation
14:54:40 sean-k-mooney gibi: o/
14:55:03 opendevreview Dan Smith proposed openstack/nova master: Log VIF event wait times https://review.opendev.org/c/openstack/nova/+/817030
14:55:10 dansmith sean-k-mooney: ^
14:55:15 sean-k-mooney kevko: at least presently since i have not backported the change to delegate taht to os-vif yet
14:56:05 sean-k-mooney oh you are using the Stopwatch functionality instead of time.now and subtracting ya that is nicer
14:56:36 kevko sean-k-mooney: what does it mean ? :/
14:56:45 kevko checking bad logs ?
14:57:10 sean-k-mooney kevko: that was to dansmith he has already created a patch to log the elapsed time
14:57:21 sean-k-mooney kevko: https://review.opendev.org/c/openstack/nova/+/817030/1/nova/compute/manager.py#491
14:57:45 sean-k-mooney kevko: oh you ment about libvirt plugging the interafce not os-vif
14:57:58 kevko yeah
14:58:54 sean-k-mooney in wallaby when we call plug on os-vif for port wiht hybrid-plug=false, such as when using the ovs firewall, os-vif just ensure the ovs bridge exists and the port is actully added to the bridge by libvirt
14:59:15 sean-k-mooney so the port wont be added until libvirt tries to start the vm
15:00:01 sean-k-mooney the neutron agent only starts wiring up the port after its created by libvirt but in this code path it will happen only after libvirt starts the vm
15:00:15 sean-k-mooney in the pasued state
15:00:58 sean-k-mooney if you were usign a differnt network backedn the behaivor would be differnt in nova
15:01:13 sean-k-mooney well potentially in any case which is why i asked
15:03:37 sean-k-mooney with your configuration the port will only be created at this point https://github.com/openstack/nova/blob/400d25fdeb45fe53be1069996ffaa3783eb4402b/nova/virt/libvirt/driver.py#L7209-L7212
15:03:53 sean-k-mooney for other configurtion it would happen eairler here https://github.com/openstack/nova/blob/400d25fdeb45fe53be1069996ffaa3783eb4402b/nova/virt/libvirt/driver.py#L7205
15:05:13 sean-k-mooney kevko: if creating the guest took a long time its possible that we woudl time out waiting for the even but that is a low proablity
15:06:38 sean-k-mooney kevko: if you put the neutron l2 agent in debug mode it will print the addtion and removal of the port in the ovsdb and also message as it configures them
15:06:51 sean-k-mooney *debug log level
15:06:54 kevko hmm, so issue is not in workers ..
15:07:08 sean-k-mooney kevko: likely not
15:07:48 kevko ok, i will try to turn on neutron debug log and see
15:07:58 sean-k-mooney kevko: i would suggest putting noa and neutron into debug mode on the host then booting another vm and check if while its waitign the domain is succesfully created in libvirt and the tap device is added to ovs
15:08:22 sean-k-mooney if it is then you shoudl check the agent log to see if it deteched it and started to process it
15:16:54 kevko thank you sean-k-mooney, probably i will ask you tomorrow
15:17:27 sean-k-mooney lyarwood: by the way im just stack a clean devstack then im going to work on the qemu wapper that we discussed last week. ill ping you when i have something
15:21:19 opendevreview Elod Illes proposed openstack/nova stable/stein: Reject open redirection in the console proxy https://review.opendev.org/c/openstack/nova/+/802935
15:29:01 lyarwood sean-k-mooney: ack sounds good
15:29:19 opendevreview Elod Illes proposed openstack/nova stable/stein: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/817037
15:56:37 dansmith sean-k-mooney: gmann: So, going back to last week's discussion about extra specs...
15:57:11 dansmith sounds like system reader and project admin should be able to see extra specs at this point to be the most compatible with existing stuff yeah?
15:57:39 dansmith (i.e. any system user can see them, only admin on the project side can see them)
15:58:51 sean-k-mooney i think project reader shoudl be able to see extra specs
15:59:04 sean-k-mooney i dont think you should need project admin
15:59:15 dansmith I know you do, but today regular users can't right?
15:59:22 sean-k-mooney they can
15:59:40 sean-k-mooney its systrem_reader_or_project_reader i think today
15:59:56 dansmith oh, maybe that explains why the test is so weird
16:00:07 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/policies/flavor_extra_specs.py#L78
16:00:29 dansmith heh yeah okay
16:00:46 sean-k-mooney adding/removing obviously should be system scoped i think
16:01:06 dansmith the test conflates a lot of stuff, I guess for that reason, so I'll have to split things apart a bit because system can't see the servers, and thus can't see the embedded flavor
16:01:29 sean-k-mooney ah right
16:12:25 dansmith hrm,
16:12:43 dansmith pretty sure there's a bug in the create test that is testing index perms for create
16:13:32 dansmith https://github.com/openstack/nova/blob/171138146a648d22474b7021ac730e26f03455f8/nova/tests/unit/policies/test_flavor_extra_specs.py#L235-L254
16:13:43 dansmith the test purports to check update,
16:13:52 melwitt gibi: ack, will look
16:13:58 dansmith but it's actually checking index, which is rule_name instead of rule
16:14:13 dansmith and it's stubbing out update for everybody instead of index
16:14:32 opendevreview Merged openstack/nova stable/victoria: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810914
16:15:16 dansmith and the test actually asserts that system reader can update extra specs
16:18:28 lyarwood elodilles / gibi ; https://review.opendev.org/q/I26b2a14e0b91c0ab77299c3e4fbed5f7916fe8cf do either of you recall why we don't need this on >= stable/victoria ?
16:21:26 lyarwood seeing some weird behaviour downstream where we hit the 2to3 issues on py39 thanks to virtualenv and setuptools versions but upstream it looks like UC is downgrading setuptools for us during the run
16:22:09 kashyap frickler: lyarwood: Unrelated: just for info: MichalP from libvirt has completed both things: switching to '-accel' by default, and wiring up tb-cache. I tested his v2 series, and looks good: https://listman.redhat.com/archives/libvir-list/2021-November/msg00236.html
16:23:41 lyarwood awesome thanks
16:31:53 elodilles lyarwood: i think it is related to victoria and newer runs over focal on gate and we haven't encounter the use_2to3 errors there
16:36:04 lyarwood elodilles: interesting, I get the feeling this might have something to do with the pip version tbh
16:36:26 lyarwood elodilles: the weirdness downstream where we don't downgrade setuptools during the run as we do upstream that is
16:53:04 gmann dansmith: we need to have separate policy now for this. for listing via 1. GET /flavors/{flavor_id}/os-extra_specs/ - system-reader-or-project-reader 2. showing extraspec in GET /servers APIs response we need project_reader 3. PUT/rebuild /servers we need to add project_member
16:53:17 dansmith gmann: yeah, but I think the existing tests are wrong
16:53:21 elodilles lyarwood: so the issue in nova was with the l-c job as it used decorator==3.4.0, which uses the use_2to3 from setuptools. In victoria the l-c.txt has decorator==4.1.0 set, which is not having the use_2to3 anymore
16:53:23 gmann is it?
16:53:23 dansmith gmann: for both create and update
16:53:46 dansmith gmann: if not I need help understanding this: https://github.com/openstack/nova/blob/171138146a648d22474b7021ac730e26f03455f8/nova/tests/unit/policies/test_flavor_extra_specs.py#L235-L254
16:53:58 dansmith gmann: we're checking update, but we're making policy for update be @
16:54:17 dansmith gmann: and we're running the check against index
16:54:40 dansmith if I run the check against update and don't stub update with @, I get a fail
16:56:26 lyarwood elodilles: sorry was on a call
16:56:35 gmann dansmith: yeah, so we are checking this policy https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/flavor_manage.py#L124
16:56:44 lyarwood elodilles: https://bugs.launchpad.net/designate/+bug/1946340 is the issue that we are hitting downstream, same root cause as the decorator problem just a different package
16:56:45 gmann dansmith: so first policy check of update has to be @
16:57:23 gmann dansmith: so that we allow update policy for everyone and see if flavor extraspec in flavor update API response is included as per extra spec policy
16:57:23 lyarwood elodilles: and with stable/wallaby upstream we don't appear to be hitting it because our upper constraints correctly downgrades setuptools during a run
16:57:28 dansmith gmann: ...but then you're just asserting that the user can run index, not that update is properly checking the thing right?
16:57:37 lyarwood elodilles: I'm just lost as to why this isn't happening downstream
16:58:04 dansmith gmann: the test is asserting that project reader can create/update flavor extra specs
16:58:14 johnsom lyarwood We removed the EOL driver that needed suds-jurko from Designate: https://review.opendev.org/c/openstack/designate/+/813380
16:58:49 gmann dansmith: this test is for 'updating flavor return the extra specs if policy allow'
16:58:50 johnsom stable branches will need to be pinned

Earlier   Later