| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-02 | |||
| 17:38:30 | sean-k-mooney | so im not sure why its different | |
| 17:38:46 | sean-k-mooney | perhaps permissions but im using hte same user on both so im not sure | |
| 17:44:15 | sean-k-mooney | lrwxrwxrwx 1 root root 49 Feb 2 17:02 devstack-int.pem -> /usr/local/share/ca-certificates/devstack-int.crt | |
| 17:44:16 | sean-k-mooney | lrwxrwxrwx 1 root root 50 Feb 2 17:02 devstack-root.pem -> /usr/local/share/ca-certificates/devstack-root.crt | |
| 17:44:28 | sean-k-mooney | thos are broken symlinks so i think if i just remove them | |
| 17:44:37 | sean-k-mooney | and stack it might fix its self | |
| 17:52:24 | sean-k-mooney | ah i missed one | |
| 17:52:36 | sean-k-mooney | i also need /opt/stack/data/devstack-cert.pem | |
| 17:53:17 | sean-k-mooney | i coppied /opt/stack/data/ca-bundle.pem and /opt/stack/data/CA/ | |
| 17:53:27 | sean-k-mooney | now it stacked fine | |
| 18:10:56 | sean-k-mooney | melwitt: https://termbin.com/ba59 less successful this time | |
| 18:11:55 | sean-k-mooney | thats using https://review.opendev.org/c/openstack/nova/+/889912 | |
| 18:14:19 | sean-k-mooney | so it looks like there is an error here https://review.opendev.org/c/openstack/nova/+/873675/22/nova/virt/libvirt/blockinfo.py#405 | |
| 18:15:18 | sean-k-mooney | my guess is you remvoed a fucntion or did not do a git add at the right time when rebasing | |
| 18:34:10 | bauzas | sean-k-mooney: noonedeadpunk: fwiw, I'm getting the same issue with my own multinode devstack I just installed | |
| 18:34:25 | sean-k-mooney | discover hosts worked fine for me | |
| 18:34:25 | bauzas | discover_hosts only works if I use --by-service | |
| 18:35:11 | sean-k-mooney | i had no issue with it using melwitt serise | |
| 18:35:24 | sean-k-mooney | i wasnt tecnially runing master but close enough | |
| 18:37:29 | bauzas | this was bizarre, the first compute service was working but I wasn't having any host_mappings values | |
| 18:41:00 | bauzas | oh wait, I found why https://paste.opendev.org/show/br3ghOvqs7NRQAfY80HX/ | |
| 18:41:28 | sean-k-mooney | oh you nit the centos libvirt bug | |
| 18:41:50 | sean-k-mooney | ya so libvirt is borked on centos for 2 days | |
| 18:41:55 | bauzas | so the service ref was creating but not the compute node one | |
| 18:42:10 | bauzas | sean-k-mooney: which one ? | |
| 18:42:21 | sean-k-mooney | that unicode error | |
| 18:42:32 | sean-k-mooney | UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 0: invalid start byt | |
| 18:42:35 | sean-k-mooney | that form libvirt | |
| 18:42:44 | sean-k-mooney | we have a cix for the next gen installer | |
| 18:42:54 | sean-k-mooney | it started seeing that in ci 2 days ago | |
| 18:43:23 | sean-k-mooney | https://issues.redhat.com/browse/OSPCIX-176 | |
| 18:43:37 | sean-k-mooney | that should be public for everyone ? | |
| 18:44:18 | clarkb | it asks me to login | |
| 18:44:35 | sean-k-mooney | ok the ci escalation project must be private... | |
| 18:45:16 | sean-k-mooney | the cijob is runign in rdo based on github patches https://review.rdoproject.org/zuul/build/5cf66b0bd0b6402faaa3306d3b193f81 | |
| 18:47:01 | bauzas | the RHEL bug is telling about the VDP issue, but I don't have this hardware | |
| 18:47:20 | sean-k-mooney | https://logserver.rdoproject.org/27/127/e7992da92f6f67327ffbf593a64b712e36b04cc6/github-check/tcib-podified-multinode-edpm-deployment-crc/5cf66b0/controller/ci-framework-data/logs/192.168.122.100/log/containers/nova/nova-compute.log | |
| 18:47:36 | sean-k-mooney | i commented that is looks like the same vpd parsing issue | |
| 18:47:45 | sean-k-mooney | but it might now be | |
| 18:48:04 | sean-k-mooney | basiclaly the libvirt api is returning non unicode bytes | |
| 18:48:16 | sean-k-mooney | so we can decode them into a strign properly | |
| 18:48:54 | sean-k-mooney | we are calling self.get_connection().getType() | |
| 18:49:10 | sean-k-mooney | which is exploding in the libvirt python module | |
| 18:49:15 | sean-k-mooney | when it calls libvirtmod.virConnectGetType(self._o) | |
| 18:49:46 | sean-k-mooney | so that a libvirt or libvirt python bug | |
| 18:49:51 | bauzas | that said I had no problem with the other compute | |
| 18:50:03 | bauzas | lemme look if I use the same version between the two computes | |
| 18:50:27 | sean-k-mooney | if your not can you record which one works | |
| 18:51:16 | bauzas | nope, I use the same version :( | |
| 18:51:26 | bauzas | that's bizarre | |
| 18:52:46 | sean-k-mooney | bauzas: can you commet on the downstream bug tracker | |
| 18:53:21 | sean-k-mooney | lets see if we can get some of our virt team folks to have a look on monday before it starts impacting the upstream gate | |
| 18:53:23 | bauzas | the CIX one ? | |
| 18:53:26 | sean-k-mooney | ya | |
| 18:53:30 | bauzas | cool | |
| 18:53:42 | sean-k-mooney | so it sound like it something works and somethime does not | |
| 18:59:37 | bauzas | actually, I can confirm that the other compute got the same exception, hence why I had no compute node | |
| 18:59:55 | sean-k-mooney | oh ok so both failed | |
| 19:00:53 | sean-k-mooney | melwitt: i fixed the missing fucntion but then it failed elsewher eon the comptue with AttributeError: 'Qcow2' object has no attribute 'disk_encryption' | |
| 19:01:35 | sean-k-mooney | in disk.libvirt_info() | |
| 19:01:40 | bauzas | apparently, now the call works for the first node | |
| 19:02:28 | sean-k-mooney | melwitt: ill try and take a look at your code again on monday if you can fix the rebase issues | |
| 19:02:36 | sean-k-mooney | bauzas: so its flaky? | |
| 19:02:51 | sean-k-mooney | i.e. on the same node it sometiems works and somethimes does not | |
| 19:03:26 | bauzas | actually, when directly calling libvirt with the python binding, I have the problem | |
| 19:03:35 | bauzas | (for the host that now works) | |
| 19:03:51 | bauzas | https://paste.opendev.org/show/blEFSbHapI8eCPEMRgyY/ | |
| 19:05:03 | sean-k-mooney | if you had a standalone repoducer that would proably help | |
| 19:05:25 | sean-k-mooney | i.e. just a python script that import libvirtmod and calls virtConnectGetType | |
| 19:05:44 | bauzas | that's what I did | |
| 19:05:57 | sean-k-mooney | oh you didnt use the nova code | |
| 19:06:19 | sean-k-mooney | ok well if you can repoduce jsut with the libvirt module in a short script | |
| 19:06:28 | sean-k-mooney | it will be a lot simpler to report this to the libvirt folks | |
| 19:06:44 | sean-k-mooney | we dont have a libvirt bug for this yet | |
| 19:07:54 | bauzas | just added a comment | |
| 19:08:01 | bauzas | lemme see if virsh gets that too | |
| 19:08:20 | sean-k-mooney | im not sure what the virsh equivlent command would be | |
| 19:17:34 | bauzas | at least domcapabilities doesn't say anything | |
| 19:18:13 | melwitt | sean-k-mooney: ugh, ok, sorry about that :( | |
| 19:19:47 | bauzas | ok I need to leave | |
| 19:20:37 | bauzas | eventually I was able to start both of the services | |
| 19:20:50 | bauzas | but we still have the bug | |
| 19:20:51 | melwitt | sean-k-mooney: every time I splice commits apart I manage to f something up 🙄 I'll get it fixed. thanks for trying it out | |
| 19:20:57 | sean-k-mooney | bauzas: i filed a bug with the libvirt folks | |
| 19:21:05 | bauzas | cool | |
| 19:24:21 | sean-k-mooney | melwitt: its fine that happens to me too | |
| 19:24:49 | artom | dansmith, I think with your stable node UUID series https://bugs.launchpad.net/nova/+bug/1817833 is fixed, no? | |
| 19:25:00 | sean-k-mooney | melwitt: i usuallly end up going commit by commmint and running a subset of tox -e py3,functional,pep8 | |
| 19:25:02 | artom | I happened upon its functional reproducer doing the downstream backport... | |
| 19:28:17 | melwitt | sean-k-mooney: yeah. I do that too ... most of the time 😒 | |
| 19:29:34 | dansmith | artom: um, not sure | |
| 19:30:19 | dansmith | artom: they seem to be complaining that they can't delete the compute to get it to have a new identity and my thing doesn't really fix that, it just means nova-compute will fail to start | |
| 19:30:28 | sean-k-mooney | artom: dansmith my readin was maybe in some specifica casses as a side effect | |
| 19:30:38 | dansmith | if you reset the uuid it will probably be right back to this problem when the name on the RP conflicts | |
| 19:31:23 | sean-k-mooney | so in the fucntionl tests the repoduce might be impacted becasue the agent will fail to start or simialr | |
| 19:31:54 | dansmith | right, it doesn't solve the problem (if you agree it's a problem) it just refuses to fail at the place it did before, and does so earlier (at start) | |
| 19:32:30 | sean-k-mooney | ya that was my feeling but i didnt fully parse the bug details | |
| 19:33:07 | artom | I wasn't really pushing one way or another, just wanted to check that we don't have stable bug reports laying around | |
| 19:33:19 | artom | (I mean, I'm sure we do, but if we can close _one_... ;) | |
| 19:34:50 | opendevreview | Merged openstack/nova stable/2023.1: Revert "[pwmgmt]ignore missin governor when cpu_state used" https://review.opendev.org/c/openstack/nova/+/905674 | |