Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-06-30
15:07:04 sean-k-mooney frickler: there is also likely one in the db decorator
15:07:20 sean-k-mooney we can see transaction error in the log
15:07:42 frickler is this with neutron-segment enabled? we have some issue with segment ID reuse in OSC
15:08:34 sean-k-mooney i think segments were enabled yes
15:08:45 frickler oh, that's the osc job, yes
15:09:28 sean-k-mooney its using vxlan however i think
15:09:39 sean-k-mooney rather then routed provider networks
15:10:07 sean-k-mooney Bound port: a2fb8af2-d4df-4b29-bd3f-5591aa8819d2, host: ubuntu-focal-rax-dfw-0030231262, vif_type: ovs, vif_details: {"connectivity": "l2", "port_filter": true, "ovs_hybrid_plug": false, "datapath_type": "system", "bridge_name": "br-int"}, binding_levels: [{'bound_driver': 'openvswitch', 'bound_segment': {'id': 'cd5c5c6b-1027-4fc7-bbc7-b8204df12e32', 'network_type': 'vxlan',
15:10:09 sean-k-mooney 'physical_network': None, 'segmentation_id': 1, 'network_id': 'ff960d9f-3b68-4b9b-8d69-78fe6441f27b'}}] {{(pid=90353) _bind_port_level /opt/stack/neutron/neutron/plugins/ml2/managers.py:948}}
15:10:11 frickler ah. maybe it is side effect of the segments test that runs in parallel. breaking other random tests
15:10:32 sean-k-mooney no right after ^ where the mech driver is able to bind
15:10:41 sean-k-mooney we get a concurrent bind excption
15:11:31 sean-k-mooney then neutron retires
15:11:31 frickler https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/job-output.txt#22299
15:11:34 sean-k-mooney 7.556330 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: WARNING neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Concurrent port binding operations failed on port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:35 sean-k-mooney Jun 30 11:36:27.557557 ubuntu-focal-rax-dfw-0030231262 neutron-server[90353]: INFO neutron.plugins.ml2.plugin [req-f9a5c6a8-ab26-4f1f-ab63-dd518edf32f3 req-c372ca6e-78a4-4f09-976b-c74d5f169c66 service neutron] Attempt 2 to bind port a2fb8af2-d4df-4b29-bd3f-5591aa8819d2
15:11:47 frickler look at ^^ the passing segment test right after the failure
15:11:59 frickler I'm pretty sure this is related
15:12:14 sean-k-mooney unless you are using the same port in both tests i dont see how it would be
15:12:24 frickler the segment test locking the DB leading to retries in other actions
15:12:59 sean-k-mooney oh so you think this trace https://zuul.opendev.org/t/openstack/build/b5c09ce1dbdd42228f5f2928d9df6178/log/controller/logs/screen-q-svc.txt#10722
15:13:05 sean-k-mooney is caused by the segment test
15:13:40 sean-k-mooney ORM session: SQL execution without transaction in progress, traceback
15:14:17 sean-k-mooney lets check the nova logs and see if there is a rety on our side
15:14:27 sean-k-mooney if not then its an internal neutron issue
15:14:53 stephenfin sean-k-mooney: I'm not sure if that's related to this issue or not
15:16:50 stephenfin actually no, maybe it is. It's an update_port call that's causing the issue
15:31:49 opendevreview Merged openstack/nova stable/xena: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833411
16:04:55 opendevreview Merged openstack/os-vif master: Check for hybrid plugging in OVS https://review.opendev.org/c/openstack/os-vif/+/839102
16:37:44 opendevreview Merged openstack/nova master: ignore deleted server groups in validation https://review.opendev.org/c/openstack/nova/+/847001
16:54:38 opendevreview Merged openstack/nova stable/wallaby: compute: Ensure updates to bdms during pre_live_migration are saved https://review.opendev.org/c/openstack/nova/+/843680
16:54:45 opendevreview Merged openstack/nova stable/wallaby: fup: Make connection_info returned by CinderFixture unique per attachment https://review.opendev.org/c/openstack/nova/+/844594
16:54:51 opendevreview Merged openstack/nova stable/wallaby: fup: Assert state of connection_info during LM rollback in func tests https://review.opendev.org/c/openstack/nova/+/844595
18:44:31 opendevreview Jay Faulkner proposed openstack/nova stable/victoria: [ironic] Minimize window for a resource provider to be lost https://review.opendev.org/c/openstack/nova/+/800873
#openstack-nova - 2022-07-01
00:33:21 opendevreview melanie witt proposed openstack/nova master: libvirt: Report ephemeral encryption traits based on imagebackend https://review.opendev.org/c/openstack/nova/+/826753
06:55:42 kashyap Has anyone ever tested live migration with vTPM?
07:47:07 opendevreview Balazs Gibizer proposed openstack/nova master: Revert "zuul: Temporarly put Centos9 Stream job non-voting" https://review.opendev.org/c/openstack/nova/+/848352
07:47:21 gibi bauzas: ^^ I think we can turn it back on
08:15:09 bauzas gibi: I have some concerns with the Centos 9S job I wanted to discuss on the last meeting
08:15:19 gibi we can keep the revert open til next week
08:15:27 bauzas thanks
08:15:36 gibi I just noticed neutron reverted the skip so I thought I propose the revert to not to forget it
08:15:55 bauzas gibi: I'll add it into the agenda
08:15:59 gibi ack
10:57:41 manuvakery1 Hi .. user-data is not getting applied for any instance which is rebuild using a snapshot taken from the same instance
10:58:06 manuvakery1 reported a bug in https://bugs.launchpad.net/nova/+bug/1980511
11:04:33 sean-k-mooney that looks like a vlaid bug your runing openstack train
11:05:10 sean-k-mooney although i wonder
11:05:46 sean-k-mooney manuvakery1: before you snapshot the instnace did you remove the file that cloud init creates to indicate that it not a first boot
11:06:22 sean-k-mooney if you did not do that then it may not have downloaded an updated copy form teh metadata api
11:06:43 sean-k-mooney manuvakery1: are you using config drive ro jsut the metadata api?
11:12:27 manuvakery1 I am using metadata api
11:13:35 manuvakery1 sean-k-mooney: i have not removed any file which cloud-init created
11:14:40 sean-k-mooney ok then this is likely not a nova bug
11:15:06 sean-k-mooney i dont think cloud init will redonwload the user data file after first boot
11:16:37 sean-k-mooney manuvakery1: https://cloudinit.readthedocs.io/en/latest/topics/cli.html#clean
11:17:22 manuvakery1 but when I tried to rebuild with another snapshot ( created from another server) i see the user-data is getting downloaded
11:17:58 sean-k-mooney the donwloading of the user data is not somethign that nova is invovled with other then servicing it via the metadata api
11:18:10 sean-k-mooney have you checked that the content in the metadata api is correct
11:18:56 sean-k-mooney curl http://169.254.169.254/openstack/latest/user_data
11:19:23 sean-k-mooney if that has the correct content form within the vm then this is not a nova bug
11:19:52 manuvakery1 ok.. I will check that
11:20:02 frickler it may be an per-instance thing, depending on whether the snapshot contains the same instance-id as the rebuilt server
11:20:31 sean-k-mooney ya that too
11:20:46 sean-k-mooney usign the snapshot on a differnt vm might work
11:20:52 sean-k-mooney but not the one it was taken form
11:21:41 sean-k-mooney the instance id is in http://169.254.169.254/2009-04-04/meta-data/instance-id
11:21:54 sean-k-mooney so its somethign the vm can discover and make desiions on
11:22:34 manuvakery1 the content in the metadata api is correct
11:23:14 sean-k-mooney then this is likely because of th instance-id as frickler said or other files created during first boot
11:27:15 manuvakery1 ok.. i will update and close the bug
11:27:39 sean-k-mooney i have it open i can do that now
11:28:37 manuvakery1 ok
11:28:37 sean-k-mooney manuvakery1: there might be a way to configure cloud init to alwasy download the user-data
11:29:58 manuvakery1 i am going to look into that right away .. let me know if any doc/links are with you
11:31:06 manuvakery1 do you think config-drive is going to make any difference?
11:31:41 sean-k-mooney https://cloudinit.readthedocs.io/en/latest/topics/boot.html?highlight=run%20on%20each%20boot#first-boot-determination
11:32:48 sean-k-mooney """By default, cloud-init attempts to determine which case it is running in by checking the instance ID in the cache against the instance ID it determines at runtime. If they do not match, then this is an instance’s first boot; otherwise, it’s a subsequent boot. """
11:35:02 sean-k-mooney manuvakery1: https://cloudinit.readthedocs.io/en/latest/topics/cli.html#single
11:35:24 sean-k-mooney you might be able to use that to make the user_data moudule run on each boot
11:36:27 manuvakery1 sean-k-mooney: i'll try that . thanks for your response
11:37:40 manuvakery1 Also i had posted another query few days back .. if i take an instance snapshot after resizing to a higher flavor the min_disk property it set to the older flavor disk size .. any hint on this
11:50:51 sean-k-mooney we filter what image porperteis are copied to the new image
11:51:00 sean-k-mooney im not sure if we have already fixed that or not
11:51:22 sean-k-mooney train is getting quite old so there has been a lot of progress over the intervening 5 releases
11:53:01 sean-k-mooney manuvakery1: you could update it via the glance api after the fact as a workaround
11:58:11 sean-k-mooney manuvakery1: ya looking at the dirver code we dont have any code to update it so it would be a new feature
12:01:21 sean-k-mooney manuvakery1: this is the relevent section fo code https://github.com/openstack/nova/blob/b597cb49a8984fbdfd8b9b4d9a692272015d83ed/nova/virt/libvirt/driver.py#L2927-L2993= but updating min_disk is not expected to happen today
12:03:34 manuvakery1 ok
12:48:44 Uggla sean-k-mooney, gibi, bauzas , I looked at instance metadata and especially devices. I attached a volume to my vm and expected to see it in the instance metadata (try to reboot the vm also). However devices field remains empty. Do I miss something ?
12:49:35 sean-k-mooney did you tag the volume
12:49:39 bauzas good question, I dunno
12:49:54 sean-k-mooney we dont generate metadta for the volumes unless you tag them
12:50:26 sean-k-mooney Uggla: https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-device-role-tagging.html
12:52:56 Uggla sean-k-mooney, ok I'm gonna try. Should we use this tagging mecanisme also with the shares ?
12:54:36 Uggla sean-k-mooney, btw congratulation for your promotion.
12:55:26 sean-k-mooney we shoudl do it automitaclly for the shares i think since we need to provide the share mount thing to the guest
12:55:38 sean-k-mooney the thing that used to mount the filesystem in teh guest
12:56:38 sean-k-mooney Uggla: also thank you i was not aware that was offically announced yet
13:05:18 sean-k-mooney oh today is july 1st so offcially the start of q3 that makes sense

Earlier   Later