| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-02 | |||
| 15:47:07 | noonedeadpunk | and I was trying to run without --by-service 10 times by now | |
| 15:47:47 | noonedeadpunk | also running nova-manage cell_v2 discover_hosts --cell_uuid UUID was just giving out "Found 0 unmapped computes in cell: UUID" | |
| 15:49:15 | noonedeadpunk | so eventually I was expecting compute to be auto-discovered... | |
| 15:49:23 | noonedeadpunk | (like all others were) | |
| 15:55:57 | bauzas | ah ok | |
| 15:56:18 | bauzas | maybe this was a bug then | |
| 16:27:08 | sean-k-mooney | noonedeadpunk: do you have the same hostname in multiple cells or perhaps is it in a diffent cell then the others | |
| 16:32:26 | noonedeadpunk | sean-k-mooney: no, not that I'm aware of - that was actuall output: https://paste.openstack.org/show/bOGJJWefzMBvhKkdHkiZ/ | |
| 16:34:12 | sean-k-mooney | hum ok ya thats odd | |
| 16:34:21 | sean-k-mooney | the names all look unique | |
| 16:34:26 | sean-k-mooney | and they are all in cell 1 | |
| 16:34:37 | noonedeadpunk | yeah, and there was no weird db records either | |
| 16:35:09 | sean-k-mooney | i think there is a verbose mode for discover_hosts but now that its fixed im usre it wont repoduce | |
| 16:35:30 | noonedeadpunk | I can imagine there could be some connectivity issues in the region as it's heavily under develpment... but I've restarted all schedulers and compute that was not found without any result | |
| 16:36:04 | noonedeadpunk | verbose was not helpful fwiw | |
| 16:36:10 | sean-k-mooney | and it worked fine when you mapped it with --by-service | |
| 16:36:14 | noonedeadpunk | https://paste.openstack.org/show/bx7p6C37Qv8LHym7Dviq/ | |
| 16:36:18 | noonedeadpunk | yup | |
| 16:36:43 | sean-k-mooney | the urllib3 thing can be ingored by the way | |
| 16:36:50 | sean-k-mooney | we should proably remove that warning | |
| 16:36:56 | noonedeadpunk | or well. I run --by-service without verbose... So I kinda assume it's what worked, as nothing else was done in between.... | |
| 16:36:57 | sean-k-mooney | it was fixed a lon gtime ago | |
| 16:38:46 | noonedeadpunk | like that's the output that was in between basically when I found it fixed: https://paste.openstack.org/show/bvFhJpJyhSv0NYn6WHaF/ | |
| 16:38:48 | sean-k-mooney | i guess if you see this again try logging with --by-service and verbose and let us know if there is anything intersting | |
| 16:39:22 | noonedeadpunk | yeah, sure... | |
| 16:39:48 | noonedeadpunk | but potentially by-service makes sense, as compute was in service... so I assume it got mapped from there then | |
| 16:39:59 | noonedeadpunk | but yeah... | |
| 16:41:46 | sean-k-mooney | i dont remmeber how that works off the top of my head so ya im not sure | |
| 16:45:35 | sean-k-mooney | ... i forgot when doing multi node devstack you need ot sync the data dir to make the ssl ca aviable if you dont turn that off | |
| 16:48:10 | sean-k-mooney | basiclaly i need to do https://opendev.org/openstack/devstack/src/branch/master/roles/sync-devstack-data/tasks/main.yaml or the compute wont be able to talk to the contoler because the ssl cert will be rejected | |
| 17:19:42 | noonedeadpunk | it's osa aio, so I SSLs should be taken care of nicely | |
| 17:20:09 | sean-k-mooney | im not refering to osa | |
| 17:20:49 | sean-k-mooney | i have have a repo where i resue the zuul job playbooks and roels to deploy multi node devstack | |
| 17:21:41 | noonedeadpunk | yeah, ok, gotcha, I guess I just missed the context | |
| 17:21:53 | sean-k-mooney | https://github.com/SeanMooney/ard/blob/6abfcae59013165404ab38ec80fa143a1c96b86a/ansible/deploy_multinode_devstack.yaml | |
| 17:22:21 | sean-k-mooney | right now im trying to figure out how to fix the ssl issues manually but might use swap to ansible | |
| 17:22:40 | sean-k-mooney | but i havent used my repo in 2 years so i dont know if it still works | |
| 17:23:01 | sean-k-mooney | i used to just turn off tls when doing multi ndoe to not need to copy the self signed certs | |
| 17:23:10 | sean-k-mooney | thats all that is currently failing | |
| 17:37:29 | clarkb | sean-k-mooney: I think you just need to copy the ca dir. IIRC devsatck doesn't centralize the ca. It makes one and then copies it so each host can use it directly | |
| 17:37:56 | sean-k-mooney | ya i did that and it didnt update the ca on the compute | |
| 17:38:15 | sean-k-mooney | the zuul jobs just copy it to the subnode before its stacks | |
| 17:38:30 | sean-k-mooney | so im not sure why its different | |
| 17:38:46 | sean-k-mooney | perhaps permissions but im using hte same user on both so im not sure | |
| 17:44:15 | sean-k-mooney | lrwxrwxrwx 1 root root 49 Feb 2 17:02 devstack-int.pem -> /usr/local/share/ca-certificates/devstack-int.crt | |
| 17:44:16 | sean-k-mooney | lrwxrwxrwx 1 root root 50 Feb 2 17:02 devstack-root.pem -> /usr/local/share/ca-certificates/devstack-root.crt | |
| 17:44:28 | sean-k-mooney | thos are broken symlinks so i think if i just remove them | |
| 17:44:37 | sean-k-mooney | and stack it might fix its self | |
| 17:52:24 | sean-k-mooney | ah i missed one | |
| 17:52:36 | sean-k-mooney | i also need /opt/stack/data/devstack-cert.pem | |
| 17:53:17 | sean-k-mooney | i coppied /opt/stack/data/ca-bundle.pem and /opt/stack/data/CA/ | |
| 17:53:27 | sean-k-mooney | now it stacked fine | |
| 18:10:56 | sean-k-mooney | melwitt: https://termbin.com/ba59 less successful this time | |
| 18:11:55 | sean-k-mooney | thats using https://review.opendev.org/c/openstack/nova/+/889912 | |
| 18:14:19 | sean-k-mooney | so it looks like there is an error here https://review.opendev.org/c/openstack/nova/+/873675/22/nova/virt/libvirt/blockinfo.py#405 | |
| 18:15:18 | sean-k-mooney | my guess is you remvoed a fucntion or did not do a git add at the right time when rebasing | |
| 18:34:10 | bauzas | sean-k-mooney: noonedeadpunk: fwiw, I'm getting the same issue with my own multinode devstack I just installed | |
| 18:34:25 | bauzas | discover_hosts only works if I use --by-service | |
| 18:34:25 | sean-k-mooney | discover hosts worked fine for me | |
| 18:35:11 | sean-k-mooney | i had no issue with it using melwitt serise | |
| 18:35:24 | sean-k-mooney | i wasnt tecnially runing master but close enough | |
| 18:37:29 | bauzas | this was bizarre, the first compute service was working but I wasn't having any host_mappings values | |
| 18:41:00 | bauzas | oh wait, I found why https://paste.opendev.org/show/br3ghOvqs7NRQAfY80HX/ | |
| 18:41:28 | sean-k-mooney | oh you nit the centos libvirt bug | |
| 18:41:50 | sean-k-mooney | ya so libvirt is borked on centos for 2 days | |
| 18:41:55 | bauzas | so the service ref was creating but not the compute node one | |
| 18:42:10 | bauzas | sean-k-mooney: which one ? | |
| 18:42:21 | sean-k-mooney | that unicode error | |
| 18:42:32 | sean-k-mooney | UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 0: invalid start byt | |
| 18:42:35 | sean-k-mooney | that form libvirt | |
| 18:42:44 | sean-k-mooney | we have a cix for the next gen installer | |
| 18:42:54 | sean-k-mooney | it started seeing that in ci 2 days ago | |
| 18:43:23 | sean-k-mooney | https://issues.redhat.com/browse/OSPCIX-176 | |
| 18:43:37 | sean-k-mooney | that should be public for everyone ? | |
| 18:44:18 | clarkb | it asks me to login | |
| 18:44:35 | sean-k-mooney | ok the ci escalation project must be private... | |
| 18:45:16 | sean-k-mooney | the cijob is runign in rdo based on github patches https://review.rdoproject.org/zuul/build/5cf66b0bd0b6402faaa3306d3b193f81 | |
| 18:47:01 | bauzas | the RHEL bug is telling about the VDP issue, but I don't have this hardware | |
| 18:47:20 | sean-k-mooney | https://logserver.rdoproject.org/27/127/e7992da92f6f67327ffbf593a64b712e36b04cc6/github-check/tcib-podified-multinode-edpm-deployment-crc/5cf66b0/controller/ci-framework-data/logs/192.168.122.100/log/containers/nova/nova-compute.log | |
| 18:47:36 | sean-k-mooney | i commented that is looks like the same vpd parsing issue | |
| 18:47:45 | sean-k-mooney | but it might now be | |
| 18:48:04 | sean-k-mooney | basiclaly the libvirt api is returning non unicode bytes | |
| 18:48:16 | sean-k-mooney | so we can decode them into a strign properly | |
| 18:48:54 | sean-k-mooney | we are calling self.get_connection().getType() | |
| 18:49:10 | sean-k-mooney | which is exploding in the libvirt python module | |
| 18:49:15 | sean-k-mooney | when it calls libvirtmod.virConnectGetType(self._o) | |
| 18:49:46 | sean-k-mooney | so that a libvirt or libvirt python bug | |
| 18:49:51 | bauzas | that said I had no problem with the other compute | |
| 18:50:03 | bauzas | lemme look if I use the same version between the two computes | |
| 18:50:27 | sean-k-mooney | if your not can you record which one works | |
| 18:51:16 | bauzas | nope, I use the same version :( | |
| 18:51:26 | bauzas | that's bizarre | |
| 18:52:46 | sean-k-mooney | bauzas: can you commet on the downstream bug tracker | |
| 18:53:21 | sean-k-mooney | lets see if we can get some of our virt team folks to have a look on monday before it starts impacting the upstream gate | |
| 18:53:23 | bauzas | the CIX one ? | |
| 18:53:26 | sean-k-mooney | ya | |
| 18:53:30 | bauzas | cool | |
| 18:53:42 | sean-k-mooney | so it sound like it something works and somethime does not | |
| 18:59:37 | bauzas | actually, I can confirm that the other compute got the same exception, hence why I had no compute node | |
| 18:59:55 | sean-k-mooney | oh ok so both failed | |