Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-17
14:04:59 bauzas or it would be "nehsse"
14:05:27 gmann i see.
14:05:43 dansmith in 'merican we don't have none of those funky looking characters
14:06:12 gmann japanese has a lot and i was always confused with those
14:06:23 dansmith uh, yeah :P
14:09:35 gmann but honestly saying using 'K' for 'c as k' sounds more clear and easy, especially for non-native english people :)
14:14:12 bauzas well, our French written level dropped a lot since a few year so I guess in a few years you'll see some Simple French too
14:14:29 bauzas which is an oxymore
14:14:35 bauzas oxymoron* actually
14:44:05 opendevreview OpenStack Release Bot proposed openstack/nova stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/nova/+/809759
14:44:10 opendevreview OpenStack Release Bot proposed openstack/nova stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/nova/+/809760
14:44:21 opendevreview OpenStack Release Bot proposed openstack/nova master: Update master for stable/xena https://review.opendev.org/c/openstack/nova/+/809761
14:44:32 opendevreview OpenStack Release Bot proposed openstack/nova master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/nova/+/809762
14:46:09 gibi \o/ we have RC1 cut
14:54:39 bauzas wahooo
15:22:30 ade_lee stephenfin, lyarwood melwitt sean-k-mooney -- hey - can anyone look at https://zuul.opendev.org/t/openstack/build/af88c0d00ebb48509a51f6eb941c65c6/logs please -- there are octavia jobs failing in a fips enabled tripleo-stanalone job here -- looks like nova api is throwing 500s
15:23:31 lyarwood ade_lee: ack I'll look shortly
15:23:44 ade_lee lyarwood, thanks!
15:26:46 sean-k-mooney interesting are there trace backs
15:28:26 sean-k-mooney looks like there was a rabit issue then we see OSError: Apache/mod_wsgi request data read error: Partial results are valid but processing is incomplete.
15:29:03 sean-k-mooney so ya there is an oslo messaging time out then a buch of thost partial result errors
15:29:10 sean-k-mooney thsi does not look related to nova
15:30:01 sean-k-mooney https://zuul.opendev.org/t/openstack/build/af88c0d00ebb48509a51f6eb941c65c6/log/logs/undercloud/var/log/containers/nova/nova-api.log#2561
15:32:28 ade_lee sean-k-mooney, hmm - guess I'll do a recheck and see if its something transient -- I see nothing in rabbit logs
15:38:41 mloza hello, I have two ceph clusters and I create cinder volumes with two types however, I can only attach first volume type to the instances, the second volume type throws an error https://paste.opendev.org/show/bq8pYJCpViYfJawKy943/
15:39:46 mloza the cinder secrets are the same on both
15:43:29 mloza I can query second ceph cluster using from the compute ceph -c ceph.conf -n client.cinder --keyring=cinder.keyring status
15:44:18 mloza not sure why nova compute is throwing perm denied
15:44:50 sean-k-mooney you likely need to add a scond keyring to the libvirt secrets
15:46:08 sean-k-mooney normally when you add ceph you do something like this https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L244-L259
15:46:53 sean-k-mooney but unless the same keyrign works for both clusters you will need to add a second one and then set the right keyring infor in the ceph backend in cinder
15:48:29 sean-k-mooney https://ceph.io/geen-categorie/openstack-nova-configure-multiple-ceph-backends-on-one-hypervisor/
15:49:04 sean-k-mooney mloza: kolla does not actuly support using muliple ceph clusters out of the box today
15:52:11 mloza sean-k-mooney: i have cinder-backup and cinder keyring the same on both ceph cluster. Yes, I modified the playbooks by hand and creating two type of volumes works.
15:53:30 mloza i can only attach first volume type two tho
15:53:38 mloza type tho*
15:53:56 sean-k-mooney you have to do somehting like this i think
15:53:58 sean-k-mooney cat <<EOF>> /etc/cinder/cinder.conf
15:54:00 sean-k-mooney [ceph]
15:54:02 sean-k-mooney volume_driver = cinder.volume.drivers.rbd.RBDDriver
15:54:04 sean-k-mooney volume_backend_name=ceph
15:54:06 sean-k-mooney rbd_pool = volumes
15:54:08 sean-k-mooney rbd_ceph_conf = /etc/ceph/ceph.conf
15:54:10 sean-k-mooney rbd_flatten_volume_from_snapshot = false
15:54:12 sean-k-mooney rbd_max_clone_depth = 5
15:54:14 sean-k-mooney rbd_store_chunk_size = 4
15:54:16 sean-k-mooney rados_connect_timeout = -1
15:54:18 sean-k-mooney glance_api_version = 2
15:54:20 sean-k-mooney rbd_user =cinder
15:54:22 sean-k-mooney rbd_secret_uuid = `cat /root/uuid`
15:54:24 sean-k-mooney EOF
15:54:25 mloza sean-k-mooney: does nova look up ceph.conf when attaching a volume?
15:54:26 sean-k-mooney 17. Restart the Cinder services.
15:54:30 sean-k-mooney no
15:54:54 sean-k-mooney we only use the info passed form cinder in the attachment connection info
15:55:02 sean-k-mooney which will incudle the rbd_secret_uuid
15:55:17 sean-k-mooney which is used to retirve the keyring form the libvirt secrte
15:56:31 sean-k-mooney mloza: the cinder folks might be able to help more
15:56:36 sean-k-mooney i know this is definetly possible
15:56:42 sean-k-mooney i jsut dont have the docs to hand
15:56:46 sean-k-mooney but other have done this
15:57:02 sean-k-mooney actully https://medium.com/walmartglobaltech/deploying-cinder-with-multiple-ceph-cluster-backends-2cd90d64b10 might be what i was looking for
15:57:46 sean-k-mooney ya that explains the addtion of the second secret and the cinder config
15:58:25 sean-k-mooney """Create the appropriate cinder user on your new Ceph cluster. The username must be different from the existing Ceph cluster cinder username"""
16:00:07 mloza i stumbled upon that blog yesterday
16:00:23 mloza looks like the second secret must be unique
16:00:41 mloza this is how my cinder.conf looks like https://paste.opendev.org/raw/809414/
16:01:40 mloza these are what inside the libvirt/secrets
16:01:45 mloza (nova-libvirt)[root@compute33 secrets]# pwd
16:01:47 mloza /etc/libvirt/secrets
16:01:49 mloza (nova-libvirt)[root@compute33 secrets]# ls -al
16:01:51 mloza total 16
16:01:53 mloza drw------- 2 root root 204 Sep 17 15:20 .
16:01:55 mloza drwxr-xr-x 1 root root 75 Sep 17 15:20 ..
16:01:57 mloza -rw------- 1 root root 40 Sep 17 15:20 1d5b8192-6acf-4f06-8c60-1eb7ecbfe911.base64
16:01:59 mloza -rw------- 1 root root 170 Sep 17 15:20 1d5b8192-6acf-4f06-8c60-1eb7ecbfe911.xml
16:02:01 mloza -rw------- 1 root root 40 Sep 17 15:20 c5730348-ff66-4491-8973-9a31317eaf00.base64
16:02:03 mloza -rw------- 1 root root 168 Sep 17 15:20 c5730348-ff66-4491-8973-9a31317eaf00.xml
16:02:05 mloza (nova-libvirt)[root@compute33 secrets]#
16:02:36 sean-k-mooney ya so your using the same user for both
16:02:50 sean-k-mooney so i think you need a different user and a diffent secret for each
16:04:33 mloza I guess the same user can stay. I'll try with a different secret
16:04:39 mloza thx for the info
16:35:02 lyarwood ade_lee: https://paste.opendev.org/show/809415/ - looks like libvirtd was super slow plugging the device for some reason
16:36:02 lyarwood assuming that's the request that octavia failed on
16:41:47 ade_lee lyarwood, thanks - lets see if it recurs
16:42:59 lyarwood ade_lee: ack, no idea how to trace the nova requests from octavia fwiw
16:45:28 johnsom lyarwood https://zuul.opendev.org/t/openstack/build/af88c0d00ebb48509a51f6eb941c65c6/log/logs/undercloud/var/log/containers/octavia/worker.log#6865
16:45:32 johnsom http://192.168.24.3:8774/v2.1/servers/52249295-f031-4f53-b403-86577a6a6e01/os-interface
16:45:51 johnsom {"interfaceAttachment": {"port_id": "47a2114d-ae49-4928-b00e-87948bd73183"}}
16:45:53 lyarwood ah ha
16:46:53 lyarwood odd that it didn't log the request-id
16:47:21 johnsom Yeah, I was puzzled too until I saw it was the tripleo haproxy process that responded with the 504
16:47:52 johnsom https://zuul.opendev.org/t/openstack/build/af88c0d00ebb48509a51f6eb941c65c6/log/logs/undercloud/var/log/containers/haproxy/haproxy.log#1276
17:33:34 opendevreview Dan Smith proposed openstack/nova master: DNM: Test glance multistore default https://review.opendev.org/c/openstack/nova/+/809883
19:50:25 mloza sean-k-mooney: i defined a different ceph user and secret for second cluster however, but i'm getting the same error https://paste.opendev.org/raw/809420/
20:02:45 mloza is rbd_secret_uuid in [libvirt] for nova.conf supports multiple uuids?
20:59:29 opendevreview melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499
20:59:29 opendevreview melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498
20:59:30 opendevreview melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271

Earlier   Later