| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-03-10 | |||
| 16:59:57 | Uggla | gibi, I'm currently try to setup devstack with ceph.... it is a nighmare... | |
| 17:00:17 | gibi | I only run the default setup with manial so no ceph there | |
| 17:02:18 | Uggla | yep the one with NFS provided on Manila site is ok. | |
| 17:04:28 | gouthamr | Uggla: o/ is something failing with devstack and manila/ceph? | |
| 17:05:31 | Uggla | gouthamr, yep it is not working with jammy, package issue and same with centos stream 9 | |
| 17:06:32 | Uggla | gouthamr, example on centos9 : ++functions-common:sudo_with_proxies:2391 sudo http_proxy= https_proxy= no_proxy= dnf install -y ceph nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rados-urls nfs-ganesha-vfs | |
| 17:06:32 | Uggla | Ceph packages for x86_64 68 kB/s | 76 kB 00:01 | |
| 17:06:32 | Uggla | Ceph noarch packages 17 kB/s | 15 kB 00:00 | |
| 17:06:32 | Uggla | NFS Ganesha packages for x86_64 14 kB/s | 13 kB 00:00 | |
| 17:06:32 | Uggla | NFS Ganesha noarch packages 1.1 kB/s | 1.0 kB 00:00 | |
| 17:06:34 | Uggla | Error: | |
| 17:06:36 | Uggla | Problem: package ceph-2:16.2.11-0.el8.x86_64 requires ceph-mgr = 2:16.2.11-0.el8, but none of the providers can be installed | |
| 17:06:39 | Uggla | - cannot install the best candidate for the job | |
| 17:06:41 | Uggla | - nothing provides libpython3.6m.so.1.0()(64bit) needed by ceph-mgr-2:16.2.11-0.el8.x86_64 | |
| 17:06:43 | Uggla | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) | |
| 17:11:53 | gouthamr | ah; ty - that's frustrating - let me look at this, most likely devstack-plugin-ceph needs some updates; we're still testing with focal on the CI because packages for Jammy haven't showed up on download.ceph.com | |
| 17:12:37 | gouthamr | i haven't tried CS9-stream, i will and raise an issue with the ceph community | |
| 17:18:54 | Uggla | gouthamr, it seams i'm progressing pre installing ceph from https://buildlogs.centos.org/centos/9-stream/storage/x86_64/ceph-pacific/ | |
| 17:21:13 | Uggla | gouthamr, not sure if I'll end up with something that works but it has passed the devstack-plugin-ceph stage. | |
| 17:31:59 | jrosser | if anyone has any influence over availabilty of jammy packages at download.ceph.com it would be really really useful to get those published | |
| 17:43:14 | sean-k-mooney | jrosser: they are onder the debian folders | |
| 17:43:59 | sean-k-mooney | or at least they used to be | |
| 17:44:01 | sean-k-mooney | http://download.ceph.com/debian-pacific/dists/ | |
| 17:44:19 | jrosser | i don't beleive there are any for quincy | |
| 17:44:49 | sean-k-mooney | yas so pacifici has focal and bionic | |
| 17:44:55 | sean-k-mooney | i dont see quincy at all | |
| 17:45:04 | sean-k-mooney | oh its there | |
| 17:45:11 | sean-k-mooney | but only focal | |
| 17:45:28 | sean-k-mooney | well and bullsye but that actully debian | |
| 17:45:35 | sean-k-mooney | it looks like they only build for the lts release | |
| 17:45:43 | sean-k-mooney | and havent added support for 22.04 | |
| 17:46:30 | sean-k-mooney | i have 0 influance over that but im surpsied they have not clsoed that gap | |
| 17:47:34 | jrosser | i would like to belive it is not intentional | |
| 17:47:42 | sean-k-mooney | https://launchpad.net/ubuntu/+source/ceph | |
| 17:47:51 | sean-k-mooney | you can use the ubuntu ppa to get the downstream one | |
| 17:48:31 | sean-k-mooney | im not sure which release 17.1.0 is https://launchpad.net/ubuntu/+source/ceph/17.1.0-0ubuntu3 | |
| 17:49:31 | sean-k-mooney | its quincy https://docs.ceph.com/en/latest/releases/index.html | |
| 17:50:12 | sean-k-mooney | jrosser: so if the disto package is an option for you thats a workaroudn for now i guess | |
| 17:51:11 | jrosser | it is very nice the way they provide versioned repos at download.ceph.com | |
| 17:51:30 | sean-k-mooney | yep i prefered using those one personally too | |
| 17:51:46 | jrosser | "as an operator" I want to test some version and stick rigidly to it unless i decide to change it | |
| 17:51:48 | dansmith | I surely hope that's not going away | |
| 17:52:07 | sean-k-mooney | it looks like its still built for 20.04 | |
| 17:52:14 | jrosser | so taking the packages from ubuntu or UCA is really unappealing to me as the pinning is much harder | |
| 17:53:08 | sean-k-mooney | ya the 20.04 package might just work but i would be good to reach out to the ceph comuncity and see if it can get fixed | |
| 17:55:28 | jrosser | sean-k-mooney: btw i had an unrelated question - what would you expect to happen if you upload an aarch64 image to an x86 cloud and boot it? | |
| 17:55:52 | jrosser | becasue i was very surprised when it booted on an x86 node under emulation rather than fail | |
| 17:55:53 | dansmith | no valid host | |
| 17:56:05 | dansmith | did you set the arch on the image? | |
| 17:56:22 | jrosser | incorrectly - to 'arm' | |
| 17:56:25 | sean-k-mooney | jrosser: https://github.com/ceph/ceph/pull/49087 looks like they curerntly have som build issue due t python 3.10 | |
| 17:56:50 | dansmith | I thought we filter hosts by arch, but maybe not | |
| 17:57:10 | sean-k-mooney | jrosser: emulation was added recently | |
| 17:57:21 | dansmith | ohhh, right | |
| 17:57:29 | jrosser | from the docs i though i had to set a property to make that happen | |
| 17:57:31 | dansmith | I thought you had to opt into that | |
| 17:57:39 | clarkb | https://packages.ubuntu.com/jammy-updates/ceph is not UCA and shouldn't change | |
| 17:58:18 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/yoga/implemented/pick-guest-arch-based-on-host-arch-in-libvirt-driver.html | |
| 18:00:05 | jrosser | whilst i was looking at this i thought there might be a docs bug here https://docs.openstack.org/nova/latest/admin/scheduling.html#imagepropertiesfilter | |
| 18:00:09 | dansmith | seems like that shoudn't be easy to enable "accidentally" | |
| 18:00:37 | jrosser | 'Describes the machine architecture required by the image. Examples are i686, x86_64, arm, and ppc64.' | |
| 18:00:45 | jrosser | arm vs AARCH64 ^ there | |
| 18:01:43 | sean-k-mooney | jrosser: https://github.com/openstack/nova/blob/master/nova/objects/fields.py#L120-L171 | |
| 18:02:15 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/arch.py#L16-L65 | |
| 18:02:29 | sean-k-mooney | so you shoudl use aarch64 | |
| 18:02:56 | sean-k-mooney | for 64bit arm | |
| 18:03:15 | sean-k-mooney | in the image properies that is | |
| 18:03:28 | sean-k-mooney | for the machine type i think its the same value | |
| 18:03:30 | jrosser | it is also unclear if those are / are not the same strings defined in os-traits | |
| 18:05:32 | jrosser | i need to investigate this more next week but what would it be that steers an hw_architecture=aarch64 | |
| 18:06:08 | sean-k-mooney | hw_architecture=aarch64 should select a host that is aarch64 | |
| 18:06:23 | jrosser | even when setting that correct it's consistently choosing an x86 compute node but i'm not sure what to check regarding the arm node to check it is a valid candiate to schedule to | |
| 18:08:54 | sean-k-mooney | and you defintly dont have hw_emulation_architecture set | |
| 18:09:36 | jrosser | no - i didnt know about that until trying to find out how the vm ended up on the wrong node | |
| 18:10:00 | sean-k-mooney | looking at the code without that set it shoudl not be using emulation | |
| 18:10:09 | sean-k-mooney | unless you have virt_type=qemu? | |
| 18:11:37 | jrosser | thats set to kvm | |
| 18:12:10 | sean-k-mooney | so this lookd right https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5242-L5277 | |
| 18:12:35 | sean-k-mooney | https://github.com/openstack/nova/blob/373be3db5b7b058767ddac50ff1367725c932a84/nova/virt/libvirt/utils.py#L509-L526 | |
| 18:12:45 | sean-k-mooney | so do you have hw_architecture set | |
| 18:14:11 | sean-k-mooney | i can maybe see why it activating on the comptue side | |
| 18:14:20 | sean-k-mooney | but im not sure why its scudlign to the compute | |
| 18:15:30 | jrosser | i think currently hw_architecture is set to 'AARCH64" | |
| 18:15:38 | jrosser | so that could still be wrong | |
| 18:16:02 | sean-k-mooney | no i think we dont care about the casing but not sure | |
| 18:16:15 | sean-k-mooney | if you have that version of qemu i think it will be able to boot | |
| 18:17:19 | sean-k-mooney | my guess is you dont have the request filter enabled? | |
| 18:17:56 | sean-k-mooney | do you have teh transform_image_metadata prefilter enabled | |
| 18:18:31 | sean-k-mooney | [scheduler]image_metadata_prefilter | |
| 18:18:40 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.image_metadata_prefilter | |
| 18:18:55 | sean-k-mooney | that is what enforce the host selection | |
| 18:20:45 | sean-k-mooney | as far as i am aware emulation is enabeld by default if you have the binary avaiable | |
| 18:21:36 | sean-k-mooney | at least without enable the prefilter | |
| 18:21:49 | sean-k-mooney | we should turn that prefilter on by default | |
| 18:21:52 | jrosser | ok i will look at the prefilter | |
| 18:21:57 | sean-k-mooney | but we dont currently | |
| 18:22:06 | jrosser | i want it to not schdule at all to x86 when i have real arm nodes | |
| 22:03:09 | opendevreview | Tyler Stachecki proposed openstack/nova master: nova-scheduler: Encourage weighers to spread https://review.opendev.org/c/openstack/nova/+/876289 | |
| #openstack-nova - 2023-03-11 | |||
| 01:02:02 | gmann | artom: commented on bug as well in gerrit, this is API backward incompatible change and cannot be done without microversion. lock APi already fixed this in 2.73 and we cannot fix it for older microversion due to backward compatibility https://review.opendev.org/c/openstack/nova/+/875653 | |
| 01:06:05 | opendevreview | Ghanshyam proposed openstack/osc-placement stable/zed: Use pypi released version of placement in functional tests https://review.opendev.org/c/openstack/osc-placement/+/869753 | |