Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-06-30
15:00:59 opendevreview Merged openstack/nova stable/wallaby: Add a regression test for bug 1939545 https://review.opendev.org/c/openstack/nova/+/843679
15:01:22 sean-k-mooney stephenfin: basically when i was looking at this i was assumeing this happened because we retried the port bidning because neutron was slow to responed
15:01:44 sean-k-mooney and that caused a concurnet bind attempt that left it in ubound or something like that
15:02:10 stephenfin Hmm, that sounds reasonable. We saw it bubble up in the OSC tests because the server create failed. That sounds like a likely root cause though
15:02:46 sean-k-mooney if that is what is happening i woudl expect to see the retry logged somewher or see two bind attempts in the nutron log
15:02:54 sean-k-mooney ill check if i can see that
15:03:28 sean-k-mooney we do see the port had just finished binding when we got the concurrent error
15:03:29 sean-k-mooney ] Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id
15:05:10 sean-k-mooney ya it looks like there are 3 attempts to bind the port
15:05:23 frickler stephenfin: I mentioned that yesterday, it also seemed related to neutron retrying binds
15:05:26 sean-k-mooney in the neutron side the last to of which had the concurrent bind excption
15:05:45 sean-k-mooney frickler: well its actully the neutornclient retrying the bind
15:06:02 sean-k-mooney frickler: that was entirely broken in nova until somewhat recently
15:06:11 melwitt bauzas: I wanted to get your thoughts on this proposed patch to change logic in the placement audit nova-manage command, since you worked on it https://review.opendev.org/c/openstack/nova/+/844418 it seems like there is a bug in the current logic but it's not clear to me what the logic should be
15:06:23 sean-k-mooney frickler: we fixed retires about a year or so ago
15:06:30 bauzas melwitt: okay, I'll look
15:06:37 melwitt thanks
15:06:41 frickler sean-k-mooney: iiuc there is an internal retry in neutron happening now
15:07:04 sean-k-mooney frickler: there is also likely one in the db decorator
15:07:20 sean-k-mooney we can see transaction error in the log
15:07:42 frickler is this with neutron-segment enabled? we have some issue with segment ID reuse in OSC
15:08:34 sean-k-mooney i think segments were enabled yes
15:08:45 frickler oh, that's the osc job, yes
15:09:28 sean-k-mooney its using vxlan however i think
15:09:39 sean-k-mooney rather then routed provider networks
15:10:07 sean-k-mooney Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id': 'cd5c5c6b-1027-4fc7-bbc7-b8204df12e32', 'network_type': 'vxlan',
15:10:09 sean-k-mooney 'physical_network': None, 'segmentation_id': 1, 'network_id': 'ff960d9f-3b68-4b9b-8d69-78fe6441f27b'}}] {{(pid=90353) _bind_port_level /opt/stack/neutron/neutron/plugins/ml2/managers.py:948}}
15:10:11 frickler ah. maybe it is side effect of the segments test that runs in parallel. breaking other random tests
15:10:32 sean-k-mooney no right after ^ where the mech driver is able to bind
15:10:41 sean-k-mooney we get a concurrent bind excption
15:11:31 frickler https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/job-output.txt#22299
15:11:31 sean-k-mooney then neutron retires
15:11:34 sean-k-mooney 7.556330 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: WARNING neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Concurrent port binding operations failed on port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:35 sean-k-mooney Jun 30 11:36:27.557557 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: INFO neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Attempt 2 to bind port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:47 frickler look at ^^ the passing segment test right after the failure
15:11:59 frickler I'm pretty sure this is related
15:12:14 sean-k-mooney unless you are using the same port in both tests i dont see how it would be
15:12:24 frickler the segment test locking the DB leading to retries in other actions
15:12:59 sean-k-mooney oh so you think this trace https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10722
15:13:05 sean-k-mooney is caused by the segment test
15:13:40 sean-k-mooney ORM session: SQL execution without transaction in progress, traceback
15:14:17 sean-k-mooney lets check the nova logs and see if there is a rety on our side
15:14:27 sean-k-mooney if not then its an internal neutron issue
15:14:53 stephenfin sean-k-mooney: I'm not sure if that's related to this issue or not
15:16:50 stephenfin actually no, maybe it is. It's an update_port call that's causing the issue
15:31:49 opendevreview Merged openstack/nova stable/xena: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833411
16:04:55 opendevreview Merged openstack/os-vif master: Check for hybrid plugging in OVS https://review.opendev.org/c/openstack/os-vif/+/839102
16:37:44 opendevreview Merged openstack/nova master: ignore deleted server groups in validation https://review.opendev.org/c/openstack/nova/+/847001
16:54:38 opendevreview Merged openstack/nova stable/wallaby: compute: Ensure updates to bdms during pre_live_migration are saved https://review.opendev.org/c/openstack/nova/+/843680
16:54:45 opendevreview Merged openstack/nova stable/wallaby: fup: Make connection_info returned by CinderFixture unique per attachment https://review.opendev.org/c/openstack/nova/+/844594
16:54:51 opendevreview Merged openstack/nova stable/wallaby: fup: Assert state of connection_info during LM rollback in func tests https://review.opendev.org/c/openstack/nova/+/844595
18:44:31 opendevreview Jay Faulkner proposed openstack/nova stable/victoria: [ironic] Minimize window for a resource provider to be lost https://review.opendev.org/c/openstack/nova/+/800873
#openstack-nova - 2022-07-01
00:33:21 opendevreview melanie witt proposed openstack/nova master: libvirt: Report ephemeral encryption traits based on imagebackend https://review.opendev.org/c/openstack/nova/+/826753
06:55:42 kashyap Has anyone ever tested live migration with vTPM?
07:47:07 opendevreview Balazs Gibizer proposed openstack/nova master: Revert "zuul: Temporarly put Centos9 Stream job non-voting" https://review.opendev.org/c/openstack/nova/+/848352
07:47:21 gibi bauzas: ^^ I think we can turn it back on
08:15:09 bauzas gibi: I have some concerns with the Centos 9S job I wanted to discuss on the last meeting
08:15:19 gibi we can keep the revert open til next week
08:15:27 bauzas thanks
08:15:36 gibi I just noticed neutron reverted the skip so I thought I propose the revert to not to forget it
08:15:55 bauzas gibi: I'll add it into the agenda
08:15:59 gibi ack
10:57:41 manuvakery1 Hi .. user-data is not getting applied for any instance which is rebuild using a snapshot taken from the same instance
10:58:06 manuvakery1 reported a bug in https://bugs.launchpad.net/nova/+bug/1980511
11:04:33 sean-k-mooney that looks like a vlaid bug your runing openstack train
11:05:10 sean-k-mooney although i wonder
11:05:46 sean-k-mooney manuvakery1: before you snapshot the instnace did you remove the file that cloud init creates to indicate that it not a first boot
11:06:22 sean-k-mooney if you did not do that then it may not have downloaded an updated copy form teh metadata api
11:06:43 sean-k-mooney manuvakery1: are you using config drive ro jsut the metadata api?
11:12:27 manuvakery1 I am using metadata api
11:13:35 manuvakery1 sean-k-mooney: i have not removed any file which cloud-init created
11:14:40 sean-k-mooney ok then this is likely not a nova bug
11:15:06 sean-k-mooney i dont think cloud init will redonwload the user data file after first boot
11:16:37 sean-k-mooney manuvakery1: https://cloudinit.readthedocs.io/en/latest/topics/cli.html#clean
11:17:22 manuvakery1 but when I tried to rebuild with another snapshot ( created from another server) i see the user-data is getting downloaded
11:17:58 sean-k-mooney the donwloading of the user data is not somethign that nova is invovled with other then servicing it via the metadata api
11:18:10 sean-k-mooney have you checked that the content in the metadata api is correct
11:18:56 sean-k-mooney curl http://169.254.169.254/openstack/latest/user_data
11:19:23 sean-k-mooney if that has the correct content form within the vm then this is not a nova bug
11:19:52 manuvakery1 ok.. I will check that
11:20:02 frickler it may be an per-instance thing, depending on whether the snapshot contains the same instance-id as the rebuilt server
11:20:31 sean-k-mooney ya that too
11:20:46 sean-k-mooney usign the snapshot on a differnt vm might work
11:20:52 sean-k-mooney but not the one it was taken form
11:21:41 sean-k-mooney the instance id is in http://169.254.169.254/2009-04-04/meta-data/instance-id
11:21:54 sean-k-mooney so its somethign the vm can discover and make desiions on
11:22:34 manuvakery1 the content in the metadata api is correct
11:23:14 sean-k-mooney then this is likely because of th instance-id as frickler said or other files created during first boot
11:27:15 manuvakery1 ok.. i will update and close the bug
11:27:39 sean-k-mooney i have it open i can do that now
11:28:37 sean-k-mooney manuvakery1: there might be a way to configure cloud init to alwasy download the user-data
11:28:37 manuvakery1 ok
11:29:58 manuvakery1 i am going to look into that right away .. let me know if any doc/links are with you
11:31:06 manuvakery1 do you think config-drive is going to make any difference?
11:31:41 sean-k-mooney https://cloudinit.readthedocs.io/en/latest/topics/boot.html?highlight=run%20on%20each%20boot#first-boot-determination
11:32:48 sean-k-mooney """By default, cloud-init attempts to determine which case it is running in by checking the instance ID in the cache against the instance ID it determines at runtime. If they do not match, then this is an instance’s first boot; otherwise, it’s a subsequent boot. """
11:35:02 sean-k-mooney manuvakery1: https://cloudinit.readthedocs.io/en/latest/topics/cli.html#single
11:35:24 sean-k-mooney you might be able to use that to make the user_data moudule run on each boot
11:36:27 manuvakery1 sean-k-mooney: i'll try that . thanks for your response
11:37:40 manuvakery1 Also i had posted another query few days back .. if i take an instance snapshot after resizing to a higher flavor the min_disk property it set to the older flavor disk size .. any hint on this

Earlier   Later