| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-10 | |||
| 16:58:43 | Uggla | gibi, ok cool, I'll have a look | |
| 16:58:47 | Uggla | gibi, thx ! | |
| 16:58:54 | gibi | I have the intention to look more and try this series out, but you can already check these two issues to see if it make sense what I found | |
| 16:59:57 | Uggla | gibi, I'm currently try to setup devstack with ceph.... it is a nighmare... | |
| 17:00:17 | gibi | I only run the default setup with manial so no ceph there | |
| 17:02:18 | Uggla | yep the one with NFS provided on Manila site is ok. | |
| 17:04:28 | gouthamr | Uggla: o/ is something failing with devstack and manila/ceph? | |
| 17:05:31 | Uggla | gouthamr, yep it is not working with jammy, package issue and same with centos stream 9 | |
| 17:06:32 | Uggla | gouthamr, example on centos9 : ++functions-common:sudo_with_proxies:2391 sudo http_proxy= https_proxy= no_proxy= dnf install -y ceph nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rados-urls nfs-ganesha-vfs | |
| 17:06:32 | Uggla | Ceph packages for x86_64 68 kB/s | 76 kB 00:01 | |
| 17:06:32 | Uggla | Ceph noarch packages 17 kB/s | 15 kB 00:00 | |
| 17:06:32 | Uggla | NFS Ganesha packages for x86_64 14 kB/s | 13 kB 00:00 | |
| 17:06:32 | Uggla | NFS Ganesha noarch packages 1.1 kB/s | 1.0 kB 00:00 | |
| 17:06:34 | Uggla | Error: | |
| 17:06:36 | Uggla | Problem: package ceph-2:16.2.11-0.el8.x86_64 requires ceph-mgr = 2:16.2.11-0.el8, but none of the providers can be installed | |
| 17:06:39 | Uggla | - cannot install the best candidate for the job | |
| 17:06:41 | Uggla | - nothing provides libpython3.6m.so.1.0()(64bit) needed by ceph-mgr-2:16.2.11-0.el8.x86_64 | |
| 17:06:43 | Uggla | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) | |
| 17:11:53 | gouthamr | ah; ty - that's frustrating - let me look at this, most likely devstack-plugin-ceph needs some updates; we're still testing with focal on the CI because packages for Jammy haven't showed up on download.ceph.com | |
| 17:12:37 | gouthamr | i haven't tried CS9-stream, i will and raise an issue with the ceph community | |
| 17:18:54 | Uggla | gouthamr, it seams i'm progressing pre installing ceph from https://buildlogs.centos.org/centos/9-stream/storage/x86_64/ceph-pacific/ | |
| 17:21:13 | Uggla | gouthamr, not sure if I'll end up with something that works but it has passed the devstack-plugin-ceph stage. | |
| 17:31:59 | jrosser | if anyone has any influence over availabilty of jammy packages at download.ceph.com it would be really really useful to get those published | |
| 17:43:14 | sean-k-mooney | jrosser: they are onder the debian folders | |
| 17:43:59 | sean-k-mooney | or at least they used to be | |
| 17:44:01 | sean-k-mooney | http://download.ceph.com/debian-pacific/dists/ | |
| 17:44:19 | jrosser | i don't beleive there are any for quincy | |
| 17:44:49 | sean-k-mooney | yas so pacifici has focal and bionic | |
| 17:44:55 | sean-k-mooney | i dont see quincy at all | |
| 17:45:04 | sean-k-mooney | oh its there | |
| 17:45:11 | sean-k-mooney | but only focal | |
| 17:45:28 | sean-k-mooney | well and bullsye but that actully debian | |
| 17:45:35 | sean-k-mooney | it looks like they only build for the lts release | |
| 17:45:43 | sean-k-mooney | and havent added support for 22.04 | |
| 17:46:30 | sean-k-mooney | i have 0 influance over that but im surpsied they have not clsoed that gap | |
| 17:47:34 | jrosser | i would like to belive it is not intentional | |
| 17:47:42 | sean-k-mooney | https://launchpad.net/ubuntu/+source/ceph | |
| 17:47:51 | sean-k-mooney | you can use the ubuntu ppa to get the downstream one | |
| 17:48:31 | sean-k-mooney | im not sure which release 17.1.0 is https://launchpad.net/ubuntu/+source/ceph/17.1.0-0ubuntu3 | |
| 17:49:31 | sean-k-mooney | its quincy https://docs.ceph.com/en/latest/releases/index.html | |
| 17:50:12 | sean-k-mooney | jrosser: so if the disto package is an option for you thats a workaroudn for now i guess | |
| 17:51:11 | jrosser | it is very nice the way they provide versioned repos at download.ceph.com | |
| 17:51:30 | sean-k-mooney | yep i prefered using those one personally too | |
| 17:51:46 | jrosser | "as an operator" I want to test some version and stick rigidly to it unless i decide to change it | |
| 17:51:48 | dansmith | I surely hope that's not going away | |
| 17:52:07 | sean-k-mooney | it looks like its still built for 20.04 | |
| 17:52:14 | jrosser | so taking the packages from ubuntu or UCA is really unappealing to me as the pinning is much harder | |
| 17:53:08 | sean-k-mooney | ya the 20.04 package might just work but i would be good to reach out to the ceph comuncity and see if it can get fixed | |
| 17:55:28 | jrosser | sean-k-mooney: btw i had an unrelated question - what would you expect to happen if you upload an aarch64 image to an x86 cloud and boot it? | |
| 17:55:52 | jrosser | becasue i was very surprised when it booted on an x86 node under emulation rather than fail | |
| 17:55:53 | dansmith | no valid host | |
| 17:56:05 | dansmith | did you set the arch on the image? | |
| 17:56:22 | jrosser | incorrectly - to 'arm' | |
| 17:56:25 | sean-k-mooney | jrosser: https://github.com/ceph/ceph/pull/49087 looks like they curerntly have som build issue due t python 3.10 | |
| 17:56:50 | dansmith | I thought we filter hosts by arch, but maybe not | |
| 17:57:10 | sean-k-mooney | jrosser: emulation was added recently | |
| 17:57:21 | dansmith | ohhh, right | |
| 17:57:29 | jrosser | from the docs i though i had to set a property to make that happen | |
| 17:57:31 | dansmith | I thought you had to opt into that | |
| 17:57:39 | clarkb | https://packages.ubuntu.com/jammy-updates/ceph is not UCA and shouldn't change | |
| 17:58:18 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/yoga/implemented/pick-guest-arch-based-on-host-arch-in-libvirt-driver.html | |
| 18:00:05 | jrosser | whilst i was looking at this i thought there might be a docs bug here https://docs.openstack.org/nova/latest/admin/scheduling.html#imagepropertiesfilter | |
| 18:00:09 | dansmith | seems like that shoudn't be easy to enable "accidentally" | |
| 18:00:37 | jrosser | 'Describes the machine architecture required by the image. Examples are i686, x86_64, arm, and ppc64.' | |
| 18:00:45 | jrosser | arm vs AARCH64 ^ there | |
| 18:01:43 | sean-k-mooney | jrosser: https://github.com/openstack/nova/blob/master/nova/objects/fields.py#L120-L171 | |
| 18:02:15 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/arch.py#L16-L65 | |
| 18:02:29 | sean-k-mooney | so you shoudl use aarch64 | |
| 18:02:56 | sean-k-mooney | for 64bit arm | |
| 18:03:15 | sean-k-mooney | in the image properies that is | |
| 18:03:28 | sean-k-mooney | for the machine type i think its the same value | |
| 18:03:30 | jrosser | it is also unclear if those are / are not the same strings defined in os-traits | |
| 18:05:32 | jrosser | i need to investigate this more next week but what would it be that steers an hw_architecture=aarch64 | |
| 18:06:08 | sean-k-mooney | hw_architecture=aarch64 should select a host that is aarch64 | |
| 18:06:23 | jrosser | even when setting that correct it's consistently choosing an x86 compute node but i'm not sure what to check regarding the arm node to check it is a valid candiate to schedule to | |
| 18:08:54 | sean-k-mooney | and you defintly dont have hw_emulation_architecture set | |
| 18:09:36 | jrosser | no - i didnt know about that until trying to find out how the vm ended up on the wrong node | |
| 18:10:00 | sean-k-mooney | looking at the code without that set it shoudl not be using emulation | |
| 18:10:09 | sean-k-mooney | unless you have virt_type=qemu? | |
| 18:11:37 | jrosser | thats set to kvm | |
| 18:12:10 | sean-k-mooney | so this lookd right https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5242-L5277 | |
| 18:12:35 | sean-k-mooney | https://github.com/openstack/nova/blob/373be3db5b7b058767ddac50ff1367725c932a84/nova/virt/libvirt/utils.py#L509-L526 | |
| 18:12:45 | sean-k-mooney | so do you have hw_architecture set | |
| 18:14:11 | sean-k-mooney | i can maybe see why it activating on the comptue side | |
| 18:14:20 | sean-k-mooney | but im not sure why its scudlign to the compute | |
| 18:15:30 | jrosser | i think currently hw_architecture is set to 'AARCH64" | |
| 18:15:38 | jrosser | so that could still be wrong | |
| 18:16:02 | sean-k-mooney | no i think we dont care about the casing but not sure | |
| 18:16:15 | sean-k-mooney | if you have that version of qemu i think it will be able to boot | |
| 18:17:19 | sean-k-mooney | my guess is you dont have the request filter enabled? | |
| 18:17:56 | sean-k-mooney | do you have teh transform_image_metadata prefilter enabled | |
| 18:18:31 | sean-k-mooney | [scheduler]image_metadata_prefilter | |
| 18:18:40 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.image_metadata_prefilter | |
| 18:18:55 | sean-k-mooney | that is what enforce the host selection | |
| 18:20:45 | sean-k-mooney | as far as i am aware emulation is enabeld by default if you have the binary avaiable | |
| 18:21:36 | sean-k-mooney | at least without enable the prefilter | |
| 18:21:49 | sean-k-mooney | we should turn that prefilter on by default | |
| 18:21:52 | jrosser | ok i will look at the prefilter | |
| 18:21:57 | sean-k-mooney | but we dont currently | |
| 18:22:06 | jrosser | i want it to not schdule at all to x86 when i have real arm nodes | |