Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-05-05
15:49:25 dansmith sean-k-mooney was also going to get an alpine guest to try to see if it ever sees the same
15:49:41 dansmith I wish we had a cirros-bug-rhel-kernel image so we could make it a RHEL thing, but.. not sure we can
15:49:53 dansmith but since this is ubuntu host, ubuntu guest, that's probably a good place to start
15:50:01 kashyap dansmith: Aaaah, nice. Can we do an "affects kernel" reference here? /me loos
15:50:05 kashyap s/loos/looks/
15:50:13 dansmith kashyap: idk
15:50:28 kashyap Don't worry; I'll do it
15:50:33 dansmith thanks
15:51:09 kashyap By "cirros-bug-rhel-kernel" image you mean CirrOS with the RHEL kernel on it, right?
15:51:29 kashyap (Yeah, I agree; as there's more kernel-team capacity that can look at)
15:51:31 dansmith ah, I typo'd
15:51:39 dansmith I meant "cirros-but-with-the-rhel-kernel" :)
15:51:52 dansmith i.e something that can run in 128MB of ram so we could use it to repro (or not) the problem :)
15:51:57 kashyap Aah, okay, I parsed it right then
15:52:38 kashyap Yeah, I see what you mean. And _still_ the RHEL folks might decline to debug - RHEL doesn't support TCG (emulation).
15:52:58 kashyap I need to go for a doc appointment; be back later. And point noted. I hope this is not grinding the CI env to a halt
15:53:30 dansmith no, but I'm just one guy and have seen this a few times this week
15:53:48 dansmith even CI impact aside, if it can happen to real instances, that's something we should care about
15:53:57 kashyap Fair enough. "Other guys might see it more times"
15:54:06 dansmith ah, noted about TCG
15:54:37 kashyap What makes this really tricky is QEMU on KVM setup :-( "KVM on KVM" would make it so much more tractable for getting the RHEL kernel/KVM folks to look at
15:55:19 dansmith yeah, well, hard to fix that really
15:55:34 dansmith if it's really TCG related that'd be interesting I guess
15:55:39 kashyap We used to get some nodes from KVM-on-KVM setup; I forget which cloud it is
15:55:59 dansmith I understand about not debugging issues in unsupported environments, but unless it matters for this, it'd suck to ignore it because of that
15:56:01 kashyap Yeah, if we can reproduce it in an env w/ KVM-on-KVM then it's a "more real issue" (for lack of a better term)
15:56:25 kashyap (I agree)
15:59:01 clarkb we have nested virt flavors that should schedule to those that do kvm on kvm
15:59:16 clarkb you could pusha change that ran the workload a bunch of times on those flavors to see if it occurs with kvm on kvm
16:00:12 clarkb re emulation not being supported: I always find it interesting when the ability to test your software in the first place is deprioritized. It feels like openshift does this as well since you can't just run the services in a few containers and test our software integration with it anymore (you could with v3 but v4 effectively killed that but some people are working on it again)
16:00:22 dansmith clarkb: the cinder tempest plugin jobs are pretty volume and compute heavy, so just running those on that flavor might be a thing
16:00:56 dansmith clarkb: can you easy button show me how to do that?
16:03:20 clarkb dansmith: ya let me pull up the necessary info
16:03:49 clarkb I think devstack's special nodesets may make this slightly more complicated than we would hope but still doable
16:06:28 clarkb dansmith: you would define a new nodeset like https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L11-L19 changing its name to something unique and modifying its label value(s) to nested-virt labels (full list of labels can be seen at https://zuul.opendev.org/t/openstack/labels) Then modify your tempest job definitions to use that new nodeset.
16:06:30 clarkb https://opendev.org/openstack/tempest/src/branch/master/zuul.d/integrated-gate.yaml#L254
16:07:12 clarkb then if you make a lot of copies of the job definition, each with a unique name, you can add them to your pipeline definition https://opendev.org/openstack/tempest/src/branch/master/zuul.d/project.yaml#L10 and run the same thing a bunch of times in parallel
16:07:23 dansmith so I depends-on that devstack change from wherever I need to do that?
16:09:28 clarkb yes, or you can just define the new nodeset directly where you do it
16:09:41 clarkb since you are making a copy it can live in the same change that you write for this (in tempest or nova or cinder etc)
16:10:13 clarkb It just needs to follow the same basic format of the devstack nodesets because devstack has expectations about groups. You can change the nodeset name and node labels though
16:10:57 dansmith ah, okay
16:11:43 dansmith would it not be easier to just redefine all or most of our nodesets to be the nested label and then I could just run regular jobs depends-on that devstack change without needing to modify the nodeset everywhere?
16:12:30 opendevreview Artom Lifshitz proposed openstack/nova stable/wallaby: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882321
16:12:31 opendevreview Artom Lifshitz proposed openstack/nova stable/wallaby: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882322
16:16:38 clarkb yes you could do that too
16:16:55 dansmith okay, lemme try that
16:20:04 clarkb it would probably be worth a note in the commit message that you shouldn't merge that change bceuase it will severely limit how many available nodes are available for running all devstack based jobs
16:20:10 clarkb I think it is a single cloud currently
16:20:15 dansmith yep, marked as DNM
16:20:43 clarkb dansmith: oh and you need to update devstack to not force emulation by default
16:20:56 dansmith yeah I saw that
16:20:59 dansmith I guess I better do that in the devstack patch too
16:21:01 dansmith I found it already
16:21:49 clarkb looks like devstack/.zuul.yaml grep for LIBVIRT_TYPE and set to kvm
16:21:53 clarkb but you found it
16:22:08 clarkb (it is set twice I think both need modification)
16:22:16 dansmith yu[p
16:26:20 dansmith clarkb: https://review.opendev.org/c/openstack/devstack/+/882457
16:26:33 dansmith "does not match definition on master" ?
16:27:25 clarkb bah
16:27:41 clarkb I guess you will need to add definitions then with unique names
16:30:12 dansmith how would one ever change them then/
16:30:22 dansmith define new and delete the old or something?
16:30:33 clarkb yes
16:31:17 clarkb there are alternatives as well. You can define anonymous nodesets in jobs directly (without names they just appl when that job runs) or in a central unbranched repo then there is a single copy of them (project-config could serve this purpose)
16:31:29 clarkb I believe they are defined directly in devstack to make it easier for third party ci systems to use though
16:31:37 dansmith okay so I should be able to just do this in c-t-p's .zuul then
16:31:50 clarkb yes that would work
16:31:52 spatel sean-k-mooney afternoon!
16:32:05 clarkb and then you can flip the LIBVIRT_TYPE var there too I think
16:32:42 spatel If you around then i have question related nova DB cleanup stuff, my machine oom out and some VMs stuck in nova DB and not sure how to clean them out.
16:48:52 dansmith clarkb: I'm messing up something with the nodeset definition: https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/882458
16:48:56 dansmith the error message is less helpful this time
16:50:57 dansmith oh wait
16:51:11 dansmith is it because name isn't indented? must be it
17:03:11 dansmith i believe it's running as expected now, thanks clarkb
17:34:54 opendevreview Balazs Gibizer proposed openstack/nova master: [doc]Clarify devname support in pci.device_spec https://review.opendev.org/c/openstack/nova/+/882464
18:03:07 clarkb sorry I stepped out for a bit
18:07:31 dansmith clarkb: no problem I think I'm good now
18:07:46 dansmith clarkb: semi-related, am I just stupid or is it impossible to get two conditions in the opensearch query?
18:08:01 dansmith any two-condition search I do always returns no results
18:08:23 dansmith and I get a syntax error
18:08:56 dansmith oh I guess I need an "and" operator
18:11:07 clarkb I don't know I'm not really invovled in it
18:12:05 dansmith it seems like I used to be able to stumble myself into a useful query and since the upgrade I can only get really basic stuff to work
18:44:01 dansmith clarkb: so, I guess something is still wrong because lib/nova switches my requested kvm to qemu because /dev/kvm is not accessible
18:44:28 dansmith https://github.com/openstack/devstack/blob/master/lib/nova#L269
18:44:59 dansmith oh, because that job didn't select the nested-jammy label, hrm
19:36:30 spatel Any idea how to clean up orphan VMs entries in nova DB?
19:36:54 spatel I have used virsh destroy command to delete vms and now DB has entries for them but VM doesn't exist
19:40:52 dansmith spatel: virsh destroy does nothing for nova vms, nova will just try to recreate them
19:41:32 spatel Hmm, I did delete them in openstack also using nova delete command
19:42:08 dansmith that's the only way, but they remain in the database until you archive (as you noted in your mailing list post)
19:42:20 dansmith archive will only remove them if they're marked as deleted
19:43:05 spatel They are doesn't exist https://paste.opendev.org/show/b0Caj0S65vx4hBFhAyML/
19:43:32 spatel openstack hypervisor stat showing 91 vms running but openstack servers list showing only single VM
19:43:51 spatel Definitely nova DB is out of sync
19:44:30 spatel I look into nova/instances DB table and there is only single entry
19:45:14 dansmith then what's the problem?

Earlier   Later