| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-05 | |||
| 15:49:53 | dansmith | but since this is ubuntu host, ubuntu guest, that's probably a good place to start | |
| 15:50:01 | kashyap | dansmith: Aaaah, nice. Can we do an "affects kernel" reference here? /me loos | |
| 15:50:05 | kashyap | s/loos/looks/ | |
| 15:50:13 | dansmith | kashyap: idk | |
| 15:50:28 | kashyap | Don't worry; I'll do it | |
| 15:50:33 | dansmith | thanks | |
| 15:51:09 | kashyap | By "cirros-bug-rhel-kernel" image you mean CirrOS with the RHEL kernel on it, right? | |
| 15:51:29 | kashyap | (Yeah, I agree; as there's more kernel-team capacity that can look at) | |
| 15:51:31 | dansmith | ah, I typo'd | |
| 15:51:39 | dansmith | I meant "cirros-but-with-the-rhel-kernel" :) | |
| 15:51:52 | dansmith | i.e something that can run in 128MB of ram so we could use it to repro (or not) the problem :) | |
| 15:51:57 | kashyap | Aah, okay, I parsed it right then | |
| 15:52:38 | kashyap | Yeah, I see what you mean. And _still_ the RHEL folks might decline to debug - RHEL doesn't support TCG (emulation). | |
| 15:52:58 | kashyap | I need to go for a doc appointment; be back later. And point noted. I hope this is not grinding the CI env to a halt | |
| 15:53:30 | dansmith | no, but I'm just one guy and have seen this a few times this week | |
| 15:53:48 | dansmith | even CI impact aside, if it can happen to real instances, that's something we should care about | |
| 15:53:57 | kashyap | Fair enough. "Other guys might see it more times" | |
| 15:54:06 | dansmith | ah, noted about TCG | |
| 15:54:37 | kashyap | What makes this really tricky is QEMU on KVM setup :-( "KVM on KVM" would make it so much more tractable for getting the RHEL kernel/KVM folks to look at | |
| 15:55:19 | dansmith | yeah, well, hard to fix that really | |
| 15:55:34 | dansmith | if it's really TCG related that'd be interesting I guess | |
| 15:55:39 | kashyap | We used to get some nodes from KVM-on-KVM setup; I forget which cloud it is | |
| 15:55:59 | dansmith | I understand about not debugging issues in unsupported environments, but unless it matters for this, it'd suck to ignore it because of that | |
| 15:56:01 | kashyap | Yeah, if we can reproduce it in an env w/ KVM-on-KVM then it's a "more real issue" (for lack of a better term) | |
| 15:56:25 | kashyap | (I agree) | |
| 15:59:01 | clarkb | we have nested virt flavors that should schedule to those that do kvm on kvm | |
| 15:59:16 | clarkb | you could pusha change that ran the workload a bunch of times on those flavors to see if it occurs with kvm on kvm | |
| 16:00:12 | clarkb | re emulation not being supported: I always find it interesting when the ability to test your software in the first place is deprioritized. It feels like openshift does this as well since you can't just run the services in a few containers and test our software integration with it anymore (you could with v3 but v4 effectively killed that but some people are working on it again) | |
| 16:00:22 | dansmith | clarkb: the cinder tempest plugin jobs are pretty volume and compute heavy, so just running those on that flavor might be a thing | |
| 16:00:56 | dansmith | clarkb: can you easy button show me how to do that? | |
| 16:03:20 | clarkb | dansmith: ya let me pull up the necessary info | |
| 16:03:49 | clarkb | I think devstack's special nodesets may make this slightly more complicated than we would hope but still doable | |
| 16:06:28 | clarkb | dansmith: you would define a new nodeset like https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L11-L19 changing its name to something unique and modifying its label value(s) to nested-virt labels (full list of labels can be seen at https://zuul.opendev.org/t/openstack/labels) Then modify your tempest job definitions to use that new nodeset. | |
| 16:06:30 | clarkb | https://opendev.org/openstack/tempest/src/branch/master/zuul.d/integrated-gate.yaml#L254 | |
| 16:07:12 | clarkb | then if you make a lot of copies of the job definition, each with a unique name, you can add them to your pipeline definition https://opendev.org/openstack/tempest/src/branch/master/zuul.d/project.yaml#L10 and run the same thing a bunch of times in parallel | |
| 16:07:23 | dansmith | so I depends-on that devstack change from wherever I need to do that? | |
| 16:09:28 | clarkb | yes, or you can just define the new nodeset directly where you do it | |
| 16:09:41 | clarkb | since you are making a copy it can live in the same change that you write for this (in tempest or nova or cinder etc) | |
| 16:10:13 | clarkb | It just needs to follow the same basic format of the devstack nodesets because devstack has expectations about groups. You can change the nodeset name and node labels though | |
| 16:10:57 | dansmith | ah, okay | |
| 16:11:43 | dansmith | would it not be easier to just redefine all or most of our nodesets to be the nested label and then I could just run regular jobs depends-on that devstack change without needing to modify the nodeset everywhere? | |
| 16:12:30 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882321 | |
| 16:12:31 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882322 | |
| 16:16:38 | clarkb | yes you could do that too | |
| 16:16:55 | dansmith | okay, lemme try that | |
| 16:20:04 | clarkb | it would probably be worth a note in the commit message that you shouldn't merge that change bceuase it will severely limit how many available nodes are available for running all devstack based jobs | |
| 16:20:10 | clarkb | I think it is a single cloud currently | |
| 16:20:15 | dansmith | yep, marked as DNM | |
| 16:20:43 | clarkb | dansmith: oh and you need to update devstack to not force emulation by default | |
| 16:20:56 | dansmith | yeah I saw that | |
| 16:20:59 | dansmith | I guess I better do that in the devstack patch too | |
| 16:21:01 | dansmith | I found it already | |
| 16:21:49 | clarkb | looks like devstack/.zuul.yaml grep for LIBVIRT_TYPE and set to kvm | |
| 16:21:53 | clarkb | but you found it | |
| 16:22:08 | clarkb | (it is set twice I think both need modification) | |
| 16:22:16 | dansmith | yu[p | |
| 16:26:20 | dansmith | clarkb: https://review.opendev.org/c/openstack/devstack/+/882457 | |
| 16:26:33 | dansmith | "does not match definition on master" ? | |
| 16:27:25 | clarkb | bah | |
| 16:27:41 | clarkb | I guess you will need to add definitions then with unique names | |
| 16:30:12 | dansmith | how would one ever change them then/ | |
| 16:30:22 | dansmith | define new and delete the old or something? | |
| 16:30:33 | clarkb | yes | |
| 16:31:17 | clarkb | there are alternatives as well. You can define anonymous nodesets in jobs directly (without names they just appl when that job runs) or in a central unbranched repo then there is a single copy of them (project-config could serve this purpose) | |
| 16:31:29 | clarkb | I believe they are defined directly in devstack to make it easier for third party ci systems to use though | |
| 16:31:37 | dansmith | okay so I should be able to just do this in c-t-p's .zuul then | |
| 16:31:50 | clarkb | yes that would work | |
| 16:31:52 | spatel | sean-k-mooney afternoon! | |
| 16:32:05 | clarkb | and then you can flip the LIBVIRT_TYPE var there too I think | |
| 16:32:42 | spatel | If you around then i have question related nova DB cleanup stuff, my machine oom out and some VMs stuck in nova DB and not sure how to clean them out. | |
| 16:48:52 | dansmith | clarkb: I'm messing up something with the nodeset definition: https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/882458 | |
| 16:48:56 | dansmith | the error message is less helpful this time | |
| 16:50:57 | dansmith | oh wait | |
| 16:51:11 | dansmith | is it because name isn't indented? must be it | |
| 17:03:11 | dansmith | i believe it's running as expected now, thanks clarkb | |
| 17:34:54 | opendevreview | Balazs Gibizer proposed openstack/nova master: [doc]Clarify devname support in pci.device_spec https://review.opendev.org/c/openstack/nova/+/882464 | |
| 18:03:07 | clarkb | sorry I stepped out for a bit | |
| 18:07:31 | dansmith | clarkb: no problem I think I'm good now | |
| 18:07:46 | dansmith | clarkb: semi-related, am I just stupid or is it impossible to get two conditions in the opensearch query? | |
| 18:08:01 | dansmith | any two-condition search I do always returns no results | |
| 18:08:23 | dansmith | and I get a syntax error | |
| 18:08:56 | dansmith | oh I guess I need an "and" operator | |
| 18:11:07 | clarkb | I don't know I'm not really invovled in it | |
| 18:12:05 | dansmith | it seems like I used to be able to stumble myself into a useful query and since the upgrade I can only get really basic stuff to work | |
| 18:44:01 | dansmith | clarkb: so, I guess something is still wrong because lib/nova switches my requested kvm to qemu because /dev/kvm is not accessible | |
| 18:44:28 | dansmith | https://github.com/openstack/devstack/blob/master/lib/nova#L269 | |
| 18:44:59 | dansmith | oh, because that job didn't select the nested-jammy label, hrm | |
| 19:36:30 | spatel | Any idea how to clean up orphan VMs entries in nova DB? | |
| 19:36:54 | spatel | I have used virsh destroy command to delete vms and now DB has entries for them but VM doesn't exist | |
| 19:40:52 | dansmith | spatel: virsh destroy does nothing for nova vms, nova will just try to recreate them | |
| 19:41:32 | spatel | Hmm, I did delete them in openstack also using nova delete command | |
| 19:42:08 | dansmith | that's the only way, but they remain in the database until you archive (as you noted in your mailing list post) | |
| 19:42:20 | dansmith | archive will only remove them if they're marked as deleted | |
| 19:43:05 | spatel | They are doesn't exist https://paste.opendev.org/show/b0Caj0S65vx4hBFhAyML/ | |
| 19:43:32 | spatel | openstack hypervisor stat showing 91 vms running but openstack servers list showing only single VM | |
| 19:43:51 | spatel | Definitely nova DB is out of sync | |
| 19:44:30 | spatel | I look into nova/instances DB table and there is only single entry | |
| 19:45:14 | dansmith | then what's the problem? | |
| 19:45:32 | dansmith | just the running_vms count? | |
| 19:45:36 | spatel | Yes... | |