Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-05-03
17:56:32 melwitt thanks sean-k-mooney
18:10:15 opendevreview Rico Lin proposed openstack/nova-specs master: Add IOMMU device support for libvirt driver https://review.opendev.org/c/openstack/nova-specs/+/840310
21:05:22 opendevreview melanie witt proposed openstack/nova-specs master: Amend unified limits spec to explain "API limit" enforcement https://review.opendev.org/c/openstack/nova-specs/+/829413
21:10:18 sean-k-mooney melwitt: thanks for fixign the merge conflict i ment to ping you about that last week
21:11:05 melwitt sean-k-mooney: np, thanks for looking
21:12:12 sean-k-mooney i was not aware that ye had provided unified_limits_count_pcpu_as_vcpu to bridge that gap
21:13:37 sean-k-mooney melwitt: have you tought about giving a presentation on this and perhaps provider.yaml to show how unified limists can now be used to provide quota for arbitry resouces
21:14:17 sean-k-mooney you could also use network bandwith as a tie in to neutron qos as another example
21:14:27 sean-k-mooney to demonstrate the new capablity
21:15:22 sean-k-mooney from a downstrema docs poing of view i think we will want ot have several such examples but i think it would be nice to have upstream docs examples or a presentaiton on the topic at some point
21:15:37 melwitt sean-k-mooney: yes, I have thought it would be useful to present or otherwise put together a doc/slides about it
21:16:42 melwitt yeah +1
21:43:25 opendevreview sean mooney proposed openstack/nova master: enable blocked VDPA move operations https://review.opendev.org/c/openstack/nova/+/832330
23:15:42 gmann bauzas: do you want me to use the BP old name for this or is it ok to change the BP name in LP (which match with the spec file), I think I did update name in LP too but you reverted - https://review.opendev.org/c/openstack/nova-specs/+/833165/1/specs/zed/approved/server-boot-on-specific-hypervisor-with-new-rbac.rst#b10
#openstack-nova - 2022-05-04
00:45:02 opendevreview Merged openstack/nova stable/wallaby: Invalidate provider tree when compute node disappears https://review.opendev.org/c/openstack/nova/+/811807
07:59:32 opendevreview anguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort https://review.opendev.org/c/openstack/nova/+/840429
08:09:01 opendevreview anguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort https://review.opendev.org/c/openstack/nova/+/840429
08:10:05 opendevreview anguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort https://review.opendev.org/c/openstack/nova/+/840429
08:13:06 opendevreview anguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort https://review.opendev.org/c/openstack/nova/+/840429
08:23:58 opendevreview anguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort https://review.opendev.org/c/openstack/nova/+/840429
08:57:13 opendevreview Balazs Gibizer proposed openstack/nova master: Allow claiming PCI PF if child VF is unavailable https://review.opendev.org/c/openstack/nova/+/838555
08:57:48 gibi melwitt, sean-k-mooney: ^^ thanks for noting the missing test coverage, I extended the patch now
09:55:51 EugenMayer Hello. Today an instance just shutdown. I checked the audit log and see that it has been stopped by 'nobody' means, the user UUID is '-' - what could that mean?
10:24:07 bauzas gmann: as you prefer, I just want to make sure that both the LP name, the spec name and the link to the LP BP in the spec are the same :)
10:24:39 bauzas gmann: if you modify the LP name tho, the link in the yoga spec won't work, but that's fine
10:56:32 opendevreview Merged openstack/nova-specs master: Amend unified limits spec to explain "API limit" enforcement https://review.opendev.org/c/openstack/nova-specs/+/829413
11:11:57 opendevreview anguoming proposed openstack/nova master: Add catching InstanceNotFound exception when call live_migration_abort https://review.opendev.org/c/openstack/nova/+/840429
11:17:23 gibi fyi reported a gate failure https://bugs.launchpad.net/neutron/+bug/1971563 it is not super frequent (two hits in 14 days) but I needed a bug number to recheck ;)
11:24:56 sean-k-mooney ok so that in ml2/ovs with iptables
11:26:05 sean-k-mooney am both the dhcp agent and the l2 agent have to set porvisioning completed for the neutron server to send the event so maybe only one of those completed in time
11:27:13 gibi yeah I tagged neutron in the bug as I only see that the port state goes from ACTIVE -> ACTIVE on neutron side after nova started waiting for the vif-plug but I don't see the notification sending in the neutron server logs
11:27:35 sean-k-mooney rovisioning for port b6dc2b79-ed38-4907-86e2-bdff1c5a9b9f completed by entity L2.
11:27:54 sean-k-mooney so ya l2 agent completed wiring it up but not dhcp agent
11:29:50 sean-k-mooney May 03 16:47:04.331996 ubuntu-focal-ovh-bhs1-0029531414 neutron-dhcp-agent[90791]: INFO neutron.agent.dhcp.agent [None req-89d4fdba-e0f4-4778-b944-ed87f5102ff1 None None] DHCP configuration for ports {'b6dc2b79-ed38-4907-86e2-bdff1c5a9b9f'} is completed
11:29:59 sean-k-mooney https://zuul.opendev.org/t/openstack/build/518f8641b9a7419391b0f99f795f26bd/log/controller/logs/screen-q-dhcp.txt#4145
11:31:54 sean-k-mooney so ya the dhcp agent things it completed but we did not see the event on the neutron server marking provisioning complete by the dhcp agent so this looks like a bug in the dhcp agent
11:32:18 sean-k-mooney normally it compeltes first so perhaps there is a bug where it wont send the event if the port is already active or something like that
11:39:13 gibi sean-k-mooney: thanks for looking into this, could you please add these details to the bug
11:39:43 sean-k-mooney sure just responeing to a review comment and ill add them
11:40:03 gibi thank you
11:55:07 sean-k-mooney gibi: actully the dhcp agent has marked it as complete
11:55:17 sean-k-mooney in the neutron server log
12:02:46 sean-k-mooney gibi: neutron did send the event
12:02:48 sean-k-mooney INFO neutron.notifiers.nova [-] Nova event matching ['req-17e4fee0-ad06-4350-af09-1db0d331d6b5'] response: {'server_uuid': '3a81145d-d263-4e1d-8ec3-faf38fed34f2', 'name': 'network-vif-plugged', 'status': 'completed', 'tag': 'b6dc2b79-ed38-4907-86e2-bdff1c5a9b9f', 'code': 200}
12:05:18 gibi sean-k-mooney: timing doesn't add up
12:05:37 gibi nova plugged the vif at May 03 16:48:41.237538
12:05:51 sean-k-mooney we recvie it at 16:47:22
12:06:11 sean-k-mooney ya so this is proably because of the dhcp agent race
12:06:25 gibi neutron should not send the plugged event _before_ nova plugs the vif
12:06:28 sean-k-mooney i bet we dont have the config option set in neutron
12:06:56 sean-k-mooney gibi: there was a race in neutron where it would not wait for both the dhcp and l2 agent to finish
12:07:35 sean-k-mooney it was fixed by https://review.opendev.org/c/openstack/neutron/+/766277
12:07:57 sean-k-mooney although hum
12:08:02 gibi this is not a live migration
12:08:02 sean-k-mooney that ws for live migration
12:08:07 gibi this is evacuate
12:08:16 sean-k-mooney the same could happen there
12:08:39 gibi and I still not get it. Can neutron send a vif-plugged event _before_ nova even plugs the vif via os-vif?
12:08:43 sean-k-mooney let me check if that is enabled or not
12:09:00 sean-k-mooney gibi: its simple
12:09:20 sean-k-mooney the port was active on the host we are evacuating form
12:09:36 sean-k-mooney so it thinks the l2 agent is finished doign its work
12:09:47 sean-k-mooney so when the dhcp agent responds it sends the event
12:09:54 sean-k-mooney that is proably what is happening here
12:10:06 sean-k-mooney that is what happend for live migration
12:10:15 sean-k-mooney i expect the same behavior for evacuate
12:11:44 sean-k-mooney gibi: https://zuul.opendev.org/t/openstack/build/518f8641b9a7419391b0f99f795f26bd/log/controller/logs/etc/neutron/neutron_conf.txt#1322-1334
12:11:47 sean-k-mooney its disabled
12:11:57 sean-k-mooney we should try enabling that and see if it fixes the problem
12:12:59 sean-k-mooney gibi: the fix in neutron is based on the presence of migratin_to in the port profile
12:13:22 gibi I thought the expected sequence would be: 1) nova bounds the port to the target host 2) nova plugs the vif on the target host 3) neutron agents plugs the other end of the vif on the target host 4) neutron sends the vif-plugged event to nova
12:14:32 gibi but based on what you said 3) and 4) happens before 2)
12:14:35 sean-k-mooney right but what actuly happens is the l2 agent on the souce host say the port is alreay bound the dhcp agent say the dhcp configurtion is correct and then neuton sends the event
12:14:45 gibi I see,
12:14:53 sean-k-mooney https://review.opendev.org/c/openstack/neutron/+/766277/10/neutron/agent/rpc.py
12:15:39 sean-k-mooney add filterign so that we only consider updates form the host that migrating_to points too
12:15:43 gibi OK
12:16:08 gibi based on the comment in the config, the live_migration_events flag should be removed in neutron in Zed already
12:16:23 sean-k-mooney well it will be removed in zed and always enabled
12:16:29 sean-k-mooney but i dont know if they have done that yet
12:16:35 gibi OK
12:17:15 sean-k-mooney https://github.com/openstack/neutron/blob/master/neutron/conf/common.py#L182-L199=
12:17:17 sean-k-mooney still there
12:17:31 gibi OK, I will push a patch to enable that flag in the hybrid plug job
12:17:59 sean-k-mooney ack ralonsoh do you have patches to remove https://github.com/openstack/neutron/blob/master/neutron/conf/common.py#L182-L199=
12:18:17 sean-k-mooney and alway smake that the correct behvior
12:23:34 opendevreview Balazs Gibizer proposed openstack/nova master: Enable live_migration_events in nova-ovs-hybrid-plug https://review.opendev.org/c/openstack/nova/+/840446
12:23:39 gibi sean-k-mooney: ^^
12:24:04 sean-k-mooney ack just realised im meent to be on a call
12:24:08 sean-k-mooney ill take a look after
12:24:12 gibi thanks
12:24:24 gibi btw this error was pretty infrequent so we might not know if this fixes it
12:26:02 sean-k-mooney ok ya if its what i think it is its an race that we ocationly loose
12:26:27 sean-k-mooney normaly the event arrives after we start waiting but not always
12:34:24 ralonsoh sean-k-mooney, I'll do it now
12:35:06 sean-k-mooney ralonsoh: thanks no rush but better to land that earlier then late in the cycle
12:35:12 ralonsoh sean-k-mooney, btw, just to confirm: the value will be True now
12:35:19 sean-k-mooney yes
12:35:21 ralonsoh perfect
12:35:25 sean-k-mooney well you are removing the option yes

Earlier   Later