| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-05 | |||
| 08:35:25 | bauzas | gibi: ah I forgot to click 'submit' | |
| 08:35:34 | bauzas | shitty today | |
| 08:36:27 | bauzas | I could be off for 15 mins, the Tesla garage (changing my windshield due to a rock) will pass me a TMX for 10 mins :) | |
| 08:57:35 | gibi | enjoy :) | |
| 10:07:53 | opendevreview | Merged openstack/nova master: Reproduce asym NUMA mixed CPU policy bug https://review.opendev.org/c/openstack/nova/+/862686 | |
| 11:43:01 | sean-k-mooney | im off today but while i remember this i found an issue with nova unit tests yesterday when trying to setup my new laptop | |
| 11:43:31 | sean-k-mooney | some of our unit tests are calling systemctl becasue of a lack of mocks | |
| 11:43:50 | sean-k-mooney | i noticed this by runing the tests in a ubuntu:22.04 container that did not have it | |
| 11:44:27 | sean-k-mooney | ill try an repoduce on monday and file a bug or fix it depeneind on how much time i have to look at it | |
| 11:46:58 | sean-k-mooney | gibi: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0ae/862687/3/gate/nova-tox-functional-py38/0aea18b/testr_results.html there is an odd db issue in teh func test result on your patch | |
| 11:47:16 | sean-k-mooney | that often meens there is a sharing of global state somewhere | |
| 11:47:54 | sean-k-mooney | it passed in other runs so we can proably just recheck it and see if its persitent but that test might be flaky so we should kep an eye on it | |
| 11:48:33 | sean-k-mooney | the failure is not related to your change | |
| 11:54:29 | gibi | sean-k-mooney: it is tracked in https://bugs.launchpad.net/nova/+bug/2002782 | |
| 11:54:45 | sean-k-mooney | oh cool | |
| 11:55:42 | sean-k-mooney | i have seen it once or twice but not often | |
| 11:55:58 | sean-k-mooney | o/ | |
| 11:56:18 | gibi | o/ | |
| 12:14:55 | opendevreview | Franciszek Przewoźny proposed openstack/placement master: Changed /tmp/migrate-db.rc to /root/migrate-db.rc https://review.opendev.org/c/openstack/placement/+/882436 | |
| 12:14:55 | opendevreview | Franciszek Przewoźny proposed openstack/placement master: Changed /tmp/migrate-db.rc to /root/migrate-db.rc https://review.opendev.org/c/openstack/placement/+/882436 | |
| 12:26:58 | opendevreview | Franciszek Przewoźny proposed openstack/placement master: Changed /tmp/migrate-db.rc to /root/migrate-db.rc https://review.opendev.org/c/openstack/placement/+/882436 | |
| 14:11:45 | opendevreview | Dan Smith proposed openstack/nova master: Populate ComputeNode.service_id https://review.opendev.org/c/openstack/nova/+/879904 | |
| 14:11:45 | opendevreview | Dan Smith proposed openstack/nova master: Add compute_id columns to instances, migrations https://review.opendev.org/c/openstack/nova/+/879499 | |
| 14:11:46 | opendevreview | Dan Smith proposed openstack/nova master: Add dest_compute_id to Migration object https://review.opendev.org/c/openstack/nova/+/879682 | |
| 14:11:46 | opendevreview | Dan Smith proposed openstack/nova master: Add compute_id to Instance object https://review.opendev.org/c/openstack/nova/+/879500 | |
| 14:11:47 | opendevreview | Dan Smith proposed openstack/nova master: Online migrate missing Instance.compute_id fields https://review.opendev.org/c/openstack/nova/+/879905 | |
| 14:12:40 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882321 | |
| 14:12:41 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882322 | |
| 15:38:13 | dansmith | kashyap: another kernel crash this morning: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3b1/881764/9/check/cinder-tempest-plugin-basic-zed/3b198f9/testr_results.html | |
| 15:39:06 | dansmith | again, in a test that is attaching volumes, but a very different scenario than the original one.. the former was ceph backed and a full tempest run, this one is non-ceph, small set of tests for just the cinder plugin | |
| 15:39:47 | dansmith | I wonder if it would make sense to just open a kernel bug instead of hitting up the virt team? | |
| 15:47:17 | kashyap | dansmith: Hmm, so it is "fairly intermittently reproducible". I'm assuming you're meaning an upstream kernel bug? (I've filed those in the past - depending on the subsystem, they get looked at; or it just rots) | |
| 15:47:29 | kashyap | s/upstream/distro/ | |
| 15:47:55 | dansmith | kashyap: well, starting with ubuntu kernel might be reasonable, yeah | |
| 15:47:56 | kashyap | dansmith: Since the host (L1) is Ubuntu, how about we (I can do it on first thing Monday) start from filing an Ubuntu kernel bug? | |
| 15:48:00 | kashyap | Bingo | |
| 15:48:03 | dansmith | https://bugs.launchpad.net/nova/+bug/2018612 | |
| 15:48:09 | dansmith | I just filed this for nova ^ | |
| 15:48:14 | dansmith | so you can reference that if you want | |
| 15:49:11 | dansmith | kashyap: yeah that'd be great if you can do that | |
| 15:49:25 | dansmith | sean-k-mooney was also going to get an alpine guest to try to see if it ever sees the same | |
| 15:49:41 | dansmith | I wish we had a cirros-bug-rhel-kernel image so we could make it a RHEL thing, but.. not sure we can | |
| 15:49:53 | dansmith | but since this is ubuntu host, ubuntu guest, that's probably a good place to start | |
| 15:50:01 | kashyap | dansmith: Aaaah, nice. Can we do an "affects kernel" reference here? /me loos | |
| 15:50:05 | kashyap | s/loos/looks/ | |
| 15:50:13 | dansmith | kashyap: idk | |
| 15:50:28 | kashyap | Don't worry; I'll do it | |
| 15:50:33 | dansmith | thanks | |
| 15:51:09 | kashyap | By "cirros-bug-rhel-kernel" image you mean CirrOS with the RHEL kernel on it, right? | |
| 15:51:29 | kashyap | (Yeah, I agree; as there's more kernel-team capacity that can look at) | |
| 15:51:31 | dansmith | ah, I typo'd | |
| 15:51:39 | dansmith | I meant "cirros-but-with-the-rhel-kernel" :) | |
| 15:51:52 | dansmith | i.e something that can run in 128MB of ram so we could use it to repro (or not) the problem :) | |
| 15:51:57 | kashyap | Aah, okay, I parsed it right then | |
| 15:52:38 | kashyap | Yeah, I see what you mean. And _still_ the RHEL folks might decline to debug - RHEL doesn't support TCG (emulation). | |
| 15:52:58 | kashyap | I need to go for a doc appointment; be back later. And point noted. I hope this is not grinding the CI env to a halt | |
| 15:53:30 | dansmith | no, but I'm just one guy and have seen this a few times this week | |
| 15:53:48 | dansmith | even CI impact aside, if it can happen to real instances, that's something we should care about | |
| 15:53:57 | kashyap | Fair enough. "Other guys might see it more times" | |
| 15:54:06 | dansmith | ah, noted about TCG | |
| 15:54:37 | kashyap | What makes this really tricky is QEMU on KVM setup :-( "KVM on KVM" would make it so much more tractable for getting the RHEL kernel/KVM folks to look at | |
| 15:55:19 | dansmith | yeah, well, hard to fix that really | |
| 15:55:34 | dansmith | if it's really TCG related that'd be interesting I guess | |
| 15:55:39 | kashyap | We used to get some nodes from KVM-on-KVM setup; I forget which cloud it is | |
| 15:55:59 | dansmith | I understand about not debugging issues in unsupported environments, but unless it matters for this, it'd suck to ignore it because of that | |
| 15:56:01 | kashyap | Yeah, if we can reproduce it in an env w/ KVM-on-KVM then it's a "more real issue" (for lack of a better term) | |
| 15:56:25 | kashyap | (I agree) | |
| 15:59:01 | clarkb | we have nested virt flavors that should schedule to those that do kvm on kvm | |
| 15:59:16 | clarkb | you could pusha change that ran the workload a bunch of times on those flavors to see if it occurs with kvm on kvm | |
| 16:00:12 | clarkb | re emulation not being supported: I always find it interesting when the ability to test your software in the first place is deprioritized. It feels like openshift does this as well since you can't just run the services in a few containers and test our software integration with it anymore (you could with v3 but v4 effectively killed that but some people are working on it again) | |
| 16:00:22 | dansmith | clarkb: the cinder tempest plugin jobs are pretty volume and compute heavy, so just running those on that flavor might be a thing | |
| 16:00:56 | dansmith | clarkb: can you easy button show me how to do that? | |
| 16:03:20 | clarkb | dansmith: ya let me pull up the necessary info | |
| 16:03:49 | clarkb | I think devstack's special nodesets may make this slightly more complicated than we would hope but still doable | |
| 16:06:28 | clarkb | dansmith: you would define a new nodeset like https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml#L11-L19 changing its name to something unique and modifying its label value(s) to nested-virt labels (full list of labels can be seen at https://zuul.opendev.org/t/openstack/labels) Then modify your tempest job definitions to use that new nodeset. | |
| 16:06:30 | clarkb | https://opendev.org/openstack/tempest/src/branch/master/zuul.d/integrated-gate.yaml#L254 | |
| 16:07:12 | clarkb | then if you make a lot of copies of the job definition, each with a unique name, you can add them to your pipeline definition https://opendev.org/openstack/tempest/src/branch/master/zuul.d/project.yaml#L10 and run the same thing a bunch of times in parallel | |
| 16:07:23 | dansmith | so I depends-on that devstack change from wherever I need to do that? | |
| 16:09:28 | clarkb | yes, or you can just define the new nodeset directly where you do it | |
| 16:09:41 | clarkb | since you are making a copy it can live in the same change that you write for this (in tempest or nova or cinder etc) | |
| 16:10:13 | clarkb | It just needs to follow the same basic format of the devstack nodesets because devstack has expectations about groups. You can change the nodeset name and node labels though | |
| 16:10:57 | dansmith | ah, okay | |
| 16:11:43 | dansmith | would it not be easier to just redefine all or most of our nodesets to be the nested label and then I could just run regular jobs depends-on that devstack change without needing to modify the nodeset everywhere? | |
| 16:12:30 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882321 | |
| 16:12:31 | opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882322 | |
| 16:16:38 | clarkb | yes you could do that too | |
| 16:16:55 | dansmith | okay, lemme try that | |
| 16:20:04 | clarkb | it would probably be worth a note in the commit message that you shouldn't merge that change bceuase it will severely limit how many available nodes are available for running all devstack based jobs | |
| 16:20:10 | clarkb | I think it is a single cloud currently | |
| 16:20:15 | dansmith | yep, marked as DNM | |
| 16:20:43 | clarkb | dansmith: oh and you need to update devstack to not force emulation by default | |
| 16:20:56 | dansmith | yeah I saw that | |
| 16:20:59 | dansmith | I guess I better do that in the devstack patch too | |
| 16:21:01 | dansmith | I found it already | |
| 16:21:49 | clarkb | looks like devstack/.zuul.yaml grep for LIBVIRT_TYPE and set to kvm | |
| 16:21:53 | clarkb | but you found it | |
| 16:22:08 | clarkb | (it is set twice I think both need modification) | |
| 16:22:16 | dansmith | yu[p | |
| 16:26:20 | dansmith | clarkb: https://review.opendev.org/c/openstack/devstack/+/882457 | |
| 16:26:33 | dansmith | "does not match definition on master" ? | |