| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-02 | |||
| 09:24:21 | gibi | elodilles: thanks a lot | |
| 10:35:44 | opendevreview | Sylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 10:35:45 | opendevreview | Sylvain Bauza proposed openstack/nova master: Modify the mdevs in the migrate XML https://review.opendev.org/c/openstack/nova/+/904258 | |
| 12:18:48 | sean-k-mooney | gibi: elodilles: im +2w on the 2023.1 gpu backports | |
| 12:22:40 | sean-k-mooney | sorry not gpu backport power management backport | |
| 12:22:48 | sean-k-mooney | https://review.opendev.org/q/topic:%22power-mgmt-fixups%22 | |
| 12:23:54 | sean-k-mooney | melwitt: ack ill deploy that in a 2 node envionment and start testing it later today | |
| 12:26:29 | gibi | sean-k-mooney: thank you | |
| 12:29:40 | elodilles | sean-k-mooney: ack, thank you o:) | |
| 12:32:03 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | |
| 13:15:16 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | |
| 14:41:02 | opendevreview | Takashi Kajinami proposed openstack/placement master: tox: Drop envdir https://review.opendev.org/c/openstack/placement/+/907590 | |
| 14:42:54 | opendevreview | Takashi Kajinami proposed openstack/nova master: tox: Drop envdir https://review.opendev.org/c/openstack/nova/+/907591 | |
| 14:49:03 | opendevreview | Takashi Kajinami proposed openstack/osc-placement master: tox: Drop envdir https://review.opendev.org/c/openstack/osc-placement/+/907596 | |
| 14:57:07 | opendevreview | Takashi Kajinami proposed openstack/os-vif master: tox: Drop envdir https://review.opendev.org/c/openstack/os-vif/+/907604 | |
| 15:29:13 | noonedeadpunk | hey folks. I have something weird going on. In a brand new (still not production) environment I've accidentally spotted, that one of computes is not mapped to the cell | |
| 15:29:54 | noonedeadpunk | so `nova-manage cell_v2 list_hosts` and `openstack compute service list --service nova-compute` differ by 1 compute not added to cell | |
| 15:30:19 | noonedeadpunk | `nova-manage cell_v2 discover_hosts` does not discover it | |
| 15:32:05 | noonedeadpunk | no errors or weird output is seen | |
| 15:32:47 | noonedeadpunk | Or well, shceduler does have `Host mapping not found for host compute01-az2. Not tracking instance info for this host.` -> thats exactly host that's missing from the cell | |
| 15:34:01 | noonedeadpunk | only 1 cell is present justi n case | |
| 15:41:55 | noonedeadpunk | hm... seem like running nova-manage cell_v2 discover_hosts --cell_uuid <UUID> --by-service did discover the host.... | |
| 15:42:25 | noonedeadpunk | and it was `--by-service` specifically | |
| 15:43:32 | noonedeadpunk | I can recall now somebody already suggested running by-service previously :) | |
| 15:45:14 | bauzas | noonedeadpunk: as a reminder, when adding a new compute, you need to use nova-manage for telling which cell you should use for it | |
| 15:45:20 | bauzas | hence why you can't find it | |
| 15:46:39 | noonedeadpunk | well, we have `discover_hosts_in_cells_interval` set | |
| 15:47:07 | noonedeadpunk | and I was trying to run without --by-service 10 times by now | |
| 15:47:47 | noonedeadpunk | also running nova-manage cell_v2 discover_hosts --cell_uuid UUID was just giving out "Found 0 unmapped computes in cell: UUID" | |
| 15:49:15 | noonedeadpunk | so eventually I was expecting compute to be auto-discovered... | |
| 15:49:23 | noonedeadpunk | (like all others were) | |
| 15:55:57 | bauzas | ah ok | |
| 15:56:18 | bauzas | maybe this was a bug then | |
| 16:27:08 | sean-k-mooney | noonedeadpunk: do you have the same hostname in multiple cells or perhaps is it in a diffent cell then the others | |
| 16:32:26 | noonedeadpunk | sean-k-mooney: no, not that I'm aware of - that was actuall output: https://paste.openstack.org/show/bOGJJWefzMBvhKkdHkiZ/ | |
| 16:34:12 | sean-k-mooney | hum ok ya thats odd | |
| 16:34:21 | sean-k-mooney | the names all look unique | |
| 16:34:26 | sean-k-mooney | and they are all in cell 1 | |
| 16:34:37 | noonedeadpunk | yeah, and there was no weird db records either | |
| 16:35:09 | sean-k-mooney | i think there is a verbose mode for discover_hosts but now that its fixed im usre it wont repoduce | |
| 16:35:30 | noonedeadpunk | I can imagine there could be some connectivity issues in the region as it's heavily under develpment... but I've restarted all schedulers and compute that was not found without any result | |
| 16:36:04 | noonedeadpunk | verbose was not helpful fwiw | |
| 16:36:10 | sean-k-mooney | and it worked fine when you mapped it with --by-service | |
| 16:36:14 | noonedeadpunk | https://paste.openstack.org/show/bx7p6C37Qv8LHym7Dviq/ | |
| 16:36:18 | noonedeadpunk | yup | |
| 16:36:43 | sean-k-mooney | the urllib3 thing can be ingored by the way | |
| 16:36:50 | sean-k-mooney | we should proably remove that warning | |
| 16:36:56 | noonedeadpunk | or well. I run --by-service without verbose... So I kinda assume it's what worked, as nothing else was done in between.... | |
| 16:36:57 | sean-k-mooney | it was fixed a lon gtime ago | |
| 16:38:46 | noonedeadpunk | like that's the output that was in between basically when I found it fixed: https://paste.openstack.org/show/bvFhJpJyhSv0NYn6WHaF/ | |
| 16:38:48 | sean-k-mooney | i guess if you see this again try logging with --by-service and verbose and let us know if there is anything intersting | |
| 16:39:22 | noonedeadpunk | yeah, sure... | |
| 16:39:48 | noonedeadpunk | but potentially by-service makes sense, as compute was in service... so I assume it got mapped from there then | |
| 16:39:59 | noonedeadpunk | but yeah... | |
| 16:41:46 | sean-k-mooney | i dont remmeber how that works off the top of my head so ya im not sure | |
| 16:45:35 | sean-k-mooney | ... i forgot when doing multi node devstack you need ot sync the data dir to make the ssl ca aviable if you dont turn that off | |
| 16:48:10 | sean-k-mooney | basiclaly i need to do https://opendev.org/openstack/devstack/src/branch/master/roles/sync-devstack-data/tasks/main.yaml or the compute wont be able to talk to the contoler because the ssl cert will be rejected | |
| 17:19:42 | noonedeadpunk | it's osa aio, so I SSLs should be taken care of nicely | |
| 17:20:09 | sean-k-mooney | im not refering to osa | |
| 17:20:49 | sean-k-mooney | i have have a repo where i resue the zuul job playbooks and roels to deploy multi node devstack | |
| 17:21:41 | noonedeadpunk | yeah, ok, gotcha, I guess I just missed the context | |
| 17:21:53 | sean-k-mooney | https://github.com/SeanMooney/ard/blob/6abfcae59013165404ab38ec80fa143a1c96b86a/ansible/deploy_multinode_devstack.yaml | |
| 17:22:21 | sean-k-mooney | right now im trying to figure out how to fix the ssl issues manually but might use swap to ansible | |
| 17:22:40 | sean-k-mooney | but i havent used my repo in 2 years so i dont know if it still works | |
| 17:23:01 | sean-k-mooney | i used to just turn off tls when doing multi ndoe to not need to copy the self signed certs | |
| 17:23:10 | sean-k-mooney | thats all that is currently failing | |
| 17:37:29 | clarkb | sean-k-mooney: I think you just need to copy the ca dir. IIRC devsatck doesn't centralize the ca. It makes one and then copies it so each host can use it directly | |
| 17:37:56 | sean-k-mooney | ya i did that and it didnt update the ca on the compute | |
| 17:38:15 | sean-k-mooney | the zuul jobs just copy it to the subnode before its stacks | |
| 17:38:30 | sean-k-mooney | so im not sure why its different | |
| 17:38:46 | sean-k-mooney | perhaps permissions but im using hte same user on both so im not sure | |
| 17:44:15 | sean-k-mooney | lrwxrwxrwx 1 root root 49 Feb 2 17:02 devstack-int.pem -> /usr/local/share/ca-certificates/devstack-int.crt | |
| 17:44:16 | sean-k-mooney | lrwxrwxrwx 1 root root 50 Feb 2 17:02 devstack-root.pem -> /usr/local/share/ca-certificates/devstack-root.crt | |
| 17:44:28 | sean-k-mooney | thos are broken symlinks so i think if i just remove them | |
| 17:44:37 | sean-k-mooney | and stack it might fix its self | |
| 17:52:24 | sean-k-mooney | ah i missed one | |
| 17:52:36 | sean-k-mooney | i also need /opt/stack/data/devstack-cert.pem | |
| 17:53:17 | sean-k-mooney | i coppied /opt/stack/data/ca-bundle.pem and /opt/stack/data/CA/ | |
| 17:53:27 | sean-k-mooney | now it stacked fine | |
| 18:10:56 | sean-k-mooney | melwitt: https://termbin.com/ba59 less successful this time | |
| 18:11:55 | sean-k-mooney | thats using https://review.opendev.org/c/openstack/nova/+/889912 | |
| 18:14:19 | sean-k-mooney | so it looks like there is an error here https://review.opendev.org/c/openstack/nova/+/873675/22/nova/virt/libvirt/blockinfo.py#405 | |
| 18:15:18 | sean-k-mooney | my guess is you remvoed a fucntion or did not do a git add at the right time when rebasing | |
| 18:34:10 | bauzas | sean-k-mooney: noonedeadpunk: fwiw, I'm getting the same issue with my own multinode devstack I just installed | |
| 18:34:25 | sean-k-mooney | discover hosts worked fine for me | |
| 18:34:25 | bauzas | discover_hosts only works if I use --by-service | |
| 18:35:11 | sean-k-mooney | i had no issue with it using melwitt serise | |
| 18:35:24 | sean-k-mooney | i wasnt tecnially runing master but close enough | |
| 18:37:29 | bauzas | this was bizarre, the first compute service was working but I wasn't having any host_mappings values | |
| 18:41:00 | bauzas | oh wait, I found why https://paste.opendev.org/show/br3ghOvqs7NRQAfY80HX/ | |
| 18:41:28 | sean-k-mooney | oh you nit the centos libvirt bug | |
| 18:41:50 | sean-k-mooney | ya so libvirt is borked on centos for 2 days | |
| 18:41:55 | bauzas | so the service ref was creating but not the compute node one | |
| 18:42:10 | bauzas | sean-k-mooney: which one ? | |
| 18:42:21 | sean-k-mooney | that unicode error | |
| 18:42:32 | sean-k-mooney | UnicodeDecodeError: 'utf-8' codec can't decode byte 0x91 in position 0: invalid start byt | |
| 18:42:35 | sean-k-mooney | that form libvirt | |
| 18:42:44 | sean-k-mooney | we have a cix for the next gen installer | |
| 18:42:54 | sean-k-mooney | it started seeing that in ci 2 days ago | |
| 18:43:23 | sean-k-mooney | https://issues.redhat.com/browse/OSPCIX-176 | |