Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-06
10:18:47 lyarwood kashyap: awesome thanks
10:19:17 bauzas lyarwood: the e-r indexing allows to set the hits automatically
10:19:21 kashyap It's the interaction of libvirt's interface with QEMU's NBD
10:19:32 bauzas lyarwood: but when you recheck with a bug number, it does increase the counter too
10:19:37 bauzas just manually
10:19:51 lyarwood bauzas: right my point is that we are likely hitting this more than the counter shows anyway
10:20:10 bauzas correct, but fwiw like I said, i incremented the counter at least this morning
10:20:38 bauzas but we can look at logstash to see the occurences if you do feel we could have more
10:20:56 bauzas http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message%3A%5C%22Live%20Migration%20failure%3A%20internal%20error%3A%20missing%20block%20job%20data%20for%20disk%20%27vda%27%3A%20libvirt.libvirtError%3A%20internal%20error%3A%20missing%20block%20job%20data%20for%20disk%20%27vda%27%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A1&from=864000s
10:21:37 lyarwood right but if the logs aren't indexed they don't show up in logstash right?
10:21:58 lyarwood so again to my point, due to the delay we are likely missing hits of this
10:22:09 lyarwood that's all I was suggesting
10:22:10 bauzas oh shit, you're right, my bad
10:22:55 bauzas I was just thinking e-r indexing was broken, not the whole logstash munging
10:23:12 bauzas (because we had specific e-r issues in the past)
10:29:28 kashyap lyarwood: Peter points out the version 5.4.0. But weren't we supposed to be using libvirt-6.0.0, from Focal?
10:38:30 lyarwood kashyap: huh
10:38:57 gibi lyarwood: regarding https://bugs.launchpad.net/nova/+bug/1902276 and the big libvirtd.log. I don't found any meaningful log I can use as a signature for this bug. so i don't know how frequently octavia hit this. Either my logstash knowledge is not enough or logstash does not index libvirtd.log
10:38:58 openstack Launchpad bug 1902276 in OpenStack Compute (nova) "libvirtd going into a tight loop causing instances to not transition to ACTIVE" [Undecided,New]
10:39:55 lyarwood gibi: kk, I'm going to finish looking at 1901739 then I'll switch back to that
10:40:10 kashyap lyarwood: What do you mean? Yeah, that was also sorta my reaction ... 5.4 is only about a year-ish old
10:40:22 gibi now I go and ask infra about how we can help with the indexing delay of logstash
10:42:31 lyarwood kashyap: gah so our nova-live-migration job is still bionic
10:42:38 lyarwood kashyap: not sure how I missed this
10:42:53 kashyap lyarwood: Either way; Peter says 6.0.0, which uses the -blockdev infra should fix it
10:43:11 lyarwood kashyap: kk well the new zuulv3 live migration job is using that and passing
10:43:12 kashyap lyarwood: No problem; you do a lot of CI job work too; easy to miss
10:43:23 kashyap Ah-ha; good to know! Got a link?
10:43:57 lyarwood kashyap: is the fix generic or can he point to something specific just so we can document why we need to move to focal?
10:44:10 lyarwood kashyap: I'm thinking of marking the current job as non-voting while we move
10:46:08 kashyap lyarwood: Yeah, that makes sense; anything that gets us away from the "legacy" (-drive)
10:46:28 kashyap lyarwood: He doesn't have a commit hash, afraid.
10:48:10 lyarwood kashyap: kk
11:05:28 openstackgerrit Lee Yarwood proposed openstack/nova master: zuul: Merge nova-evacuate into nova-multinode-live-migration https://review.opendev.org/755309
11:05:28 openstackgerrit Lee Yarwood proposed openstack/nova master: zuul: Replace nova-live-migration with zuulv3 jobs https://review.opendev.org/752557
11:06:15 lyarwood gibi / kashyap ; ^ so I think we need to bring this forward to resolve https://bugs.launchpad.net/nova/+bug/1901739
11:06:15 openstack Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,In progress] - Assigned to Lee Yarwood (lyarwood)
11:06:19 lyarwood ahead of the ceph stuff also landing
11:06:31 lyarwood tl;dr move the live migration job from bionic to focal
11:09:07 gibi lyarwood: looking...
11:10:10 openstackgerrit Hemanth Nakkina proposed openstack/nova stable/train: Update pci stat pools based on PCI device changes https://review.opendev.org/761725
11:16:27 openstackgerrit Hemanth Nakkina proposed openstack/nova stable/train: Update pci stat pools based on PCI device changes https://review.opendev.org/761725
11:19:34 gibi lyarwood: so with the new focal job we can avoid 1901739 on master, but we still hit it on stable/victoria isn't it?
11:25:23 openstackgerrit Hemanth Nakkina proposed openstack/nova stable/stein: Update pci stat pools based on PCI device changes https://review.opendev.org/761727
11:48:03 lyarwood gibi: yes, unless we backport the job there
11:48:18 lyarwood gibi: that might not be a crazy idea
11:48:33 gibi yeah, for victoria it is OK as we had the focal goal for V
11:48:46 gibi does this bug hits older stable branches too?
11:49:03 lyarwood gibi: I think I've only seen it in master thus far
11:49:12 lyarwood gibi: but we haven't been landing much in stable/victoria
11:49:27 lyarwood gibi: vs master that is
11:49:47 openstackgerrit Hemanth Nakkina proposed openstack/nova stable/stein: Update pci stat pools based on PCI device changes https://review.opendev.org/761727
12:01:51 gibi lyarwood: ack
12:03:31 gibi I totally support backporting this job changes to V
12:06:54 lyarwood gibi: ack thanks
12:58:03 sean-k-mooney elod: can you take a look at this os-vif backport and the proceeding patch when you have time https://review.opendev.org/#/c/759255/1
13:04:06 gibi gmann: does this ring a bell to you? https://zuul.opendev.org/t/openstack/build/bf932888821345c1955cda68226b1a87/log/job-output.txt#28429 was there any policy default change in glance or policy config change in devstack recently?
13:19:12 elod sean-k-mooney: sure! (at first glance the 'refactor' topic tricked me, but as far as i see it's a bugfix :))
13:30:23 sean-k-mooney ya kind of its pulling out some common code but its need for the follow up
13:57:00 mnasiadka good afternoon
13:57:19 mnasiadka does Nova support having ephemeral volumes from different backends? (e.g. one ceph, second from a local filesystem)
14:22:57 lyarwood mnasiadka: for the libvirt virt driver no, we only support a single images_type configurable
14:23:06 lyarwood mnasiadka: for multibackend support you should use cinder
14:23:27 mnasiadka lyarwood: that's what I considered as an alternative
14:23:30 mnasiadka lyarwood: thanks
14:23:32 lyarwood mnasiadka: np
14:25:53 sean-k-mooney stephenfin:.... https://github.com/openstack/nova/commit/b8695de6da56db42b83b9d9d4c330148766644be rewrote the pci generation that is why my fix is broken
14:26:49 sean-k-mooney i guess that needs to go back to queens too
14:44:18 bauzas gibi: others, some bugfix that could be backported https://review.opendev.org/#/c/758470/
14:44:25 bauzas easy peasy IMHO
14:45:58 gibi bauzas: will check soonish
14:46:04 bauzas no rush
14:46:06 bauzas but ta
14:47:15 gibi it is Friday I run out of rush alreay ;)
14:49:08 openstackgerrit sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642
14:49:08 openstackgerrit sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432
14:52:36 lyarwood I'm not sure if anyone has been looking at this but I can't see an obvious issue with nova-ceph-multistore aside from instance spawns taking a long time overall
14:53:03 lyarwood I'll look at another example failure after a downstream call and file a bug to track these timeouts
14:54:13 gmann gibi: let me check. we did change one glance policy in nove ceph multistore job.
14:54:44 gmann gibi: ah this is same copy_image policy. checking why it is failing.
14:55:25 gibi gmann: thanks!
14:56:16 bauzas is it only me or you do you get a shit of exceptions when running functional and unittests locally with nova.storage.rbd_utils about the 'rados' and 'rbd' packages missing ?
14:56:24 bauzas or do*
14:56:27 openstackgerrit Ghanshyam Mann proposed openstack/nova stable/victoria: Modify glance's copy_image permission for nova-ceph-multistore https://review.opendev.org/761755
14:56:30 gibi bauzas: I see it too
14:56:42 gibi and I remember seeing a patch fixing it
14:56:45 bauzas I tested with both py38 and py39
14:56:54 bauzas but that's mostly annoying
14:56:55 gmann gibi: bauzas lyarwood backporting the nova-ceph-multistore fix https://review.opendev.org/#/c/761755/
14:57:13 gibi bauzas: https://review.opendev.org/#/c/746873/
14:57:15 gmann Tempest test merged before we backported this
14:58:08 gibi gmann: thanks!
14:58:13 bauzas gibi: meh, I should have it in my local repo
14:58:47 gibi bauzas: then maybe it is not the fix but it is the patch that generates the exception in the first place
14:58:58 lyarwood gmann: ah did that bork the job on stable?
14:59:21 gmann gibi: lyarwood nova-ceph-multistore job is from victoria so backporting until victoria is needed
14:59:29 gmann lyarwood: yeah - https://zuul.opendev.org/t/openstack/build/bf932888821345c1955cda68226b1a87/
14:59:59 gmann tempest test which started running in nova-ceph-multistore on master as well on stable/victoria started running and needed this change on nova side
15:00:16 bauzas gibi: hah, that's maybe because the exception is not the same
15:00:52 bauzas ah no

Earlier   Later