Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-10-01
13:24:37 lyarwood oh I wondeer
13:24:42 lyarwood interesting
13:25:01 sean-k-mooney On PCIe devices, this cancels the hot-unplug process.
13:25:13 lyarwood I don't recall seeing another request from libvirt to QEMU in the focal examples
13:25:15 lyarwood but maybe
13:25:31 sean-k-mooney if we retry it it would right
13:25:37 sean-k-mooney we just loop
13:25:48 lyarwood right but I can't recall seeing the requests overlap
13:25:59 lyarwood but the loop is pretty tight
13:26:12 lyarwood would explain why we only saw it on overloaded CI hosts
13:27:47 kashyap lyarwood: Hey, was out for a small walk; reading your comments in the change
13:28:37 kashyap That "unable to execute ... device_del' rings a bell
13:34:22 kashyap lyarwood: I think we _are_ hitting the bug the check was introduced to avoid, no? The QEMU commit skips repeated QMP 'device_del' calls
13:37:19 lyarwood kashyap: right and with >=v5.0.0 that raises an error, I was wondering above if prior to that on focal we are actually hitting the original issue and hot unplugs are being cancelled in QEMU
13:37:45 kashyap lyarwood: Ah, hence your comment on "why we only saw it on overloaded CI hosts"
13:38:01 lyarwood kashyap: either way the fix here seems to be to use libvirt events to wait until a detach either succeeds or fails before we retry
13:38:04 sean-k-mooney kashyap: right when we race with a slow detach due to load
13:38:45 sean-k-mooney lyarwood: ya so we should do that in either case and then we can use the ci to see if it fixes the ci issue with a DNM patch to swap the job to focal form bionic
13:39:16 sean-k-mooney lyarwood: we do sometime get random detach failure on bionic too
13:39:21 sean-k-mooney which could be caused by this
13:40:01 sean-k-mooney its much less common but i have seen many jobs fail for seminly unrelated volume detach issues
13:40:36 kashyap lyarwood: Yeah, it's a long-time TODO to wait for proper events, instead of polling :-( There should also be a documented TODO in driver.py/guest.py
14:23:01 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM libvirt: Remove loopingcall when detaching device https://review.opendev.org/755526
14:49:28 openstackgerrit sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642
14:49:37 openstackgerrit sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432
14:56:50 sean-k-mooney those are just rebases by the way ^ i want to see if the tempest failure is real or not
14:57:21 sean-k-mooney i dont think they are but a rebase never hurts
15:45:14 gibi nova weekly meeting starts in 15 minutes in #openstack-meeting-3
16:30:35 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
16:31:34 gibi lyarwood, stephenfin : could you hit it again? ^
16:32:32 lyarwood gibi: will do once it passes
16:32:46 gibi OK :)
16:33:05 gibi thanks
16:33:26 gibi I will leave for today in 30 minutes so I guess I won't see it pass
16:35:40 stephenfin #
16:35:55 stephenfin Done. If it doesn't pass, it simply won't merge :)
16:38:09 gibi thanks
21:09:26 openstackgerrit melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444
21:09:27 openstackgerrit melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055
21:09:28 openstackgerrit melanie witt proposed openstack/nova stable/pike: Add resource_class to fields in ironic node cache https://review.opendev.org/755653
#openstack-nova - 2020-10-02
07:41:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
08:41:18 openstackgerrit Stephen Finucane proposed openstack/nova master: api-ref: Move 'os-agents' API to obsolete section https://review.opendev.org/755729
13:44:15 lyarwood stephenfin: https://review.opendev.org/#/c/755498/ can you hit this agian please
13:46:13 stephenfin done
13:51:14 lyarwood ta
14:14:37 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Increase incremental and max sleep time during detach https://review.opendev.org/755799
14:32:22 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Increase incremental and max sleep time during detach https://review.opendev.org/755799
16:49:40 openstackgerrit Merged openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
17:01:58 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-evacuate: Disable libvirtd service and sockets during negative tests https://review.opendev.org/754684
17:02:26 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985
17:02:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986
17:17:09 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add tests for mixed CPU policy https://review.opendev.org/755852
17:17:47 stephenfin sean-k-mooney, artom: Should be an easy fix if either of you are bored :-P I'm off for a run though so will take a look next week if not https://bugs.launchpad.net/nova/+bug/1898272
17:17:47 openstack Launchpad bug 1898272 in OpenStack Compute (nova) ""mixed" policy calculations don't account for host cells with no free shared CPUs" [Medium,Confirmed] - Assigned to Stephen Finucane (stephenfinucane)
17:19:15 artom stephenfin, fuuuun
17:19:26 stephenfin you're telling me :(
17:54:19 sean-k-mooney hum ya that a bug
17:54:46 sean-k-mooney you could have moved it to triaged too
17:55:39 sean-k-mooney stephenfin: does mixed cpus support numa at all
17:56:45 sean-k-mooney we shoudl be able to test this with whitebox by the way
17:56:57 sean-k-mooney we can test it with a functional test too
17:58:01 sean-k-mooney stephenfin: thats failing because we are not tracking the numa affinity fo the shared cores yes
18:04:38 sean-k-mooney ah i see the fixt thats subtle
18:11:39 sean-k-mooney stephenfin: there is more legecy debt there
18:11:52 sean-k-mooney stephenfin: the cpu toplogy is not per numa node
18:11:54 sean-k-mooney its per vm
18:12:07 sean-k-mooney so we should not have a toplogy filed in the instance cell at all
18:13:13 sean-k-mooney so like the pagesize we should be pulling that field out of the instance numa cell object
18:14:39 sean-k-mooney both of these https://github.com/openstack/nova/blob/master/nova/objects/instance_numa.py#L79-L81 should be removed
18:15:08 sean-k-mooney and the cpu_thread_policy
18:16:11 sean-k-mooney they coudl be moved to the InstanceNUMATopology object i guess
18:16:31 sean-k-mooney although they are not really realted to numa but it would be more correct
18:16:55 sean-k-mooney stephenfin: can you add a todo to remove them
18:24:03 sean-k-mooney stephenfin: for what its worth i dont think we need the siblings field on the instance_cell object either
18:24:26 sean-k-mooney i think we only ever use teh siblings proerty on the host cell object not the property ihn the instnace cell
18:24:34 sean-k-mooney at least based on http://codesearch.openstack.org/?q=siblings&i=nope&files=&repos=openstack/nova
18:24:39 sean-k-mooney its not obvious if we do
18:29:07 sean-k-mooney the inital implemenation was quite bad at stashing things in the instnace cell object for convenince hten actully ensuign we only had one souce or truth
#openstack-nova - 2020-10-03
02:41:45 openstackgerrit melanie witt proposed openstack/nova master: Initialize global data separately and run_once in WSGI app init https://review.opendev.org/733627
04:53:07 openstackgerrit melanie witt proposed openstack/nova master: Initialize global data separately and run_once in WSGI app init https://review.opendev.org/733627
06:03:27 openstackgerrit Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175
12:18:52 openstackgerrit Lee Yarwood proposed openstack/nova master: bindep: Install python3 and python3-devel on CentOS 8 and Fedora https://review.opendev.org/755928
12:21:01 openstackgerrit Lee Yarwood proposed openstack/nova master: bindep: Install python3 and python3-devel on CentOS 8 and Fedora https://review.opendev.org/755928
#openstack-nova - 2020-10-04
18:10:49 openstackgerrit Merged openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985
18:14:57 openstackgerrit Merged openstack/nova master: nova-evacuate: Disable libvirtd service and sockets during negative tests https://review.opendev.org/754684
#openstack-nova - 2020-10-05
03:47:23 openstackgerrit Wenping Song proposed openstack/nova master: Add instance project_id for cyborg arq https://review.opendev.org/738428
07:02:02 openstackgerrit Balazs Gibizer proposed openstack/nova master: Revert "Temporarily disable parts of heal port allocation test" https://review.opendev.org/755995
08:37:22 hemanth_n gibi: thanks for the review on https://review.opendev.org/#/c/749175/ last thursday, sean have clarified on your comments and i added them as code comments to avoid confusion..
08:46:17 openstackgerrit Lee Yarwood proposed openstack/nova master: zuul: Merge nova-evacuate into nova-multinode-live-migration https://review.opendev.org/755309
08:58:29 gibi hemanth_n: I will go back to that review today
08:59:22 hemanth_n gibi: thanks
09:47:53 jiriprox Hi, I would need your support. We are using kolla images for train release from docker.io and last week on Tuesday images stopped working in our environment. PCI passthrough devices a are properly detected on compute node but they are not stored to DB. When we check corresponding table in nova db on controller it is empty :-(
10:23:48 openstackgerrit Wenping Song proposed openstack/nova master: Add instance project_id for cyborg arq https://review.opendev.org/738428
10:33:01 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701
10:39:47 gibi jiriprox: have you traced back the kolla image changes to a certain nova version change in those imageS?
10:44:13 jiriprox @gibi No. I don't know how to track which source code version was used in docker image pulled from docker.io with tag train
10:44:41 gibi jiriprox: do you know which was the last docker image version that worked for you?
10:47:31 jiriprox @gibi The last one working was pulled 25.9. It stopped working with images pulled 29.9.
10:48:11 gibi jiriprox: could you check with the kolla developers if they can map those dates to nova versions?
10:48:27 gibi sean-k-mooney: Do you happen to know such mapping ? ^^

Earlier   Later