| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-23 | |||
| 16:58:59 | haleyb | bauzas: we have timeslots on thursday, one later one on wednesday | |
| 17:00:02 | bauzas | thursday would fit | |
| 17:01:09 | haleyb | bauzas: thursday 15:00 ? | |
| 17:01:30 | bauzas | wfm | |
| 17:01:45 | bauzas | haleyb: don't you mind if we use our etherpad and our room ? | |
| 17:02:32 | haleyb | that works, i'll add the link in our etherpad | |
| 17:02:47 | bauzas | haleyb: https://etherpad.opendev.org/p/nova-caracal-ptg#L300https://etherpad.opendev.org/p/nova-caracal-ptg#L59 | |
| 17:02:50 | bauzas | doh | |
| 17:02:52 | bauzas | https://etherpad.opendev.org/p/nova-caracal-ptg#L59 | |
| 17:40:21 | gmann | bauzas: I added the Nova rbac service role discussion continuation at the bottom in etherpad, https://etherpad.opendev.org/p/nova-caracal-ptg | |
| 17:40:34 | bauzas | gmann: saw it yeah | |
| 18:23:52 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Deprecate use_cow_images and 'default' images_type https://review.opendev.org/c/openstack/nova/+/898229 | |
| 18:23:52 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Fix image conversion check in finish_migration https://review.opendev.org/c/openstack/nova/+/897842 | |
| 19:04:54 | simondodsley | Is there a way to remove a boot volume from a shutdown instance and replace it with a new boot volume (created from an earlier snapshot)? | |
| 19:11:58 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Fix image conversion check in finish_migration https://review.opendev.org/c/openstack/nova/+/897842 | |
| 19:11:59 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Deprecate use_cow_images and 'default' images_type https://review.opendev.org/c/openstack/nova/+/898229 | |
| 20:34:13 | sean-k-mooney | bauzas: there has been a very "interesting" bug that has been evolving over the last few days which i added to the ptg | |
| 20:34:33 | sean-k-mooney | bauzas: https://bugs.launchpad.net/nova/+bug/2038898 https://etherpad.opendev.org/p/nova-caracal-ptg#L444 | |
| 20:35:44 | sean-k-mooney | it woudl be good to read back over that wehn you have time and for others. the reporter has done a lot of testing at my request to determinthe corrent beahvior fo nova and its very much not what i expect | |
| 20:36:58 | sean-k-mooney | images_type=raw is still jsut an aliase fo images_type=flat and does not force convertion to raw. i was aware of th history between those two backend but i tough thwen we renamed raw to flat and added raw as an alais it woudl force image convertion but it does not | |
| 20:37:39 | sean-k-mooney | use_qow_images=true is bacially ignored if you use images_type=flat in that it never force the vm to be qcow | |
| 20:37:59 | sean-k-mooney | if you use image_type=default then use_qow_images=true request in us using image_type=qcow | |
| 20:42:38 | simondodsley | Is this spec ever going to be implemented... https://blueprints.launchpad.net/nova/+spec/detach-boot-volume | |
| 20:43:12 | sean-k-mooney | simondodsley: i dont belive anyone is working on it and i belvie there are still some desgin isues with enabling that | |
| 20:43:27 | sean-k-mooney | simondodsley: so its very unlikely | |
| 20:47:13 | simondodsley | sean-k-mooney it's a pretty fundamental requirement, especially if we want to be able to compete more with VMware. There are a lot of people looking to pivot at the moment and not having a "simple" thing like this won't help | |
| 20:50:05 | clarkb | I think you can emulate it by booting off of a new volume and moving neutron ports over / using fips? | |
| 20:50:59 | clarkb | that said we tend to avoid boot from volume as much as possible and instead prefering to attach volumes for persistent data and then let the OS be more ephemeral | |
| 21:03:00 | simondodsley | clarkb: not convinced that is the main implementation when there are highly intelligent external arrays being used under cinder. All of our customers are either only using boot from volume, or are migrating to it to get away from limitations of local nova disk, or limitations of ceph | |
| 21:04:58 | clarkb | I'm not saying its the main implementation/ I'm just giving my experience as a user. boot from volume is more problematic than the value we get out of it so we try to use volumes more explicitly for where the data lives rather than our operating systems which are actually quite ephemeral | |
| 21:05:11 | clarkb | which is similar to how containers deal with this stuff too | |
| 21:05:57 | clarkb | For example rescuing boot from volume instances is still a bit of an ordeal | |
| 21:08:11 | clarkb | I shouldtest that again actually | |
| 21:13:34 | simondodsley | using boot from volume can be highly advantagous to users as external arrays can both deduplicate and compress volumes on the array and so 100s of VMs can boot from snapshot clones of the boot image, thereby actually using only the size of the source volume on the array. | |
| 21:13:52 | simondodsley | this is why we need features like changing the boot volume to be available. | |
| 21:14:27 | simondodsley | yes, cinder can do restore from snapshot, but that is limited as well. | |
| #openstack-nova - 2023-10-24 | |||
| 01:10:28 | sean-k-mooney[m] | simondodsley: one thing to keep in mind is openstack and nova in particular is not an enterprise virtualisation platform and vmware is not a compator, aws or azure is a better comparison | |
| 01:12:50 | sean-k-mooney[m] | ignoreing the fact ware are likely to remove the vmware driver shortly, vmware is one of the backend hyperviors you can use with nova. but nova is a cloud compute project so feature parity with enterpirse virtualisation software is not a goal for nova | |
| 07:10:23 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova stable/2023.2: Install lxml before we need it in post-run https://review.opendev.org/c/openstack/nova/+/899142 | |
| 07:10:33 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova stable/2023.2: Revert "Add upgrade check for compute-object-ids linkage" https://review.opendev.org/c/openstack/nova/+/898953 | |
| 08:16:48 | bauzas | fwiw, I'm more than happy to welcome volunteers wanting to work on https://blueprints.launchpad.net/nova/+spec/detach-boot-volume in particular if they want this is important for their users | |
| 08:17:19 | bauzas | I can offer nova onboarding and knowledge transfer if that helps | |
| 08:17:48 | bauzas | I'm paid for leading a community, but I need this community to enact by itself :) | |
| 08:41:29 | bauzas | simondodsley: fwiw, we have an PTG operator hour that's proposed today at 1500UTC https://ptg.opendev.org/ptg.html | |
| 08:41:51 | bauzas | simondodsley: if you feel you'd like to discuss your feature request, feel free to join | |
| 08:42:27 | noonedeadpunk | hey folks. I get weird error on HEAD of stable/2023.1 when trying to rebuild the instance: https://paste.openstack.org/show/biUIcOzMCx0YlsFob2KK/ | |
| 08:42:29 | bauzas | and like I said, if I can help someone volunteering, I'll surely offer onboarding | |
| 08:43:02 | bauzas | noonedeadpunk: eek | |
| 08:43:14 | noonedeadpunk | I decided to ask if it's smth known already so that I won't spend time on looking deeper (as for me it's not very obvious from the trace) | |
| 08:46:32 | noonedeadpunk | yeah, detach-boot-volume is really interesting thing, though I can recall it being not as trivial as it might look at the beginning | |
| 08:46:38 | bauzas | noonedeadpunk: this is a RPC call | |
| 08:46:56 | noonedeadpunk | ok, so I should check on.. conductor I guess | |
| 08:47:19 | bauzas | this sounds a compat issue between conductor and compute | |
| 08:47:36 | bauzas | basically your compute awaits some argument | |
| 08:47:42 | bauzas | that conductor isn't passing | |
| 08:47:52 | bauzas | I guess you upgraded ? | |
| 08:48:02 | bauzas | do you have a rolling upgrade in lieu ? | |
| 08:50:04 | bauzas | noonedeadpunk: that's the client RPC code that does the RPC negociation https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L1106-L1110 | |
| 08:50:30 | bauzas | this client code is run by the nova-conductor service | |
| 08:50:42 | bauzas | or by a compute service if this is compute-to-compute calls | |
| 08:50:45 | noonedeadpunk | Yeah, I am. And I see version 66 for every service in `services` table of cell db | |
| 08:51:35 | noonedeadpunk | oh, wait, couple of computes are still 60.... Hm | |
| 08:51:55 | noonedeadpunk | Not the affected one though, but still smth I need to fix | |
| 08:52:46 | bauzas | do you have some rpc pins ? | |
| 08:54:01 | noonedeadpunk | nah, I don't | |
| 08:54:13 | noonedeadpunk | Or well, should not have - never set them | |
| 08:54:15 | bauzas | https://github.com/openstack/nova/blob/stable/2023.1/nova/conductor/manager.py#L1395-L1414 is where conductor calls compute for rebuilding | |
| 08:55:05 | noonedeadpunk | Ok, computes that have eariler version are down right now... | |
| 08:55:09 | bauzas | the conductor manager is then invoking compute.rpcapi's rebuild_instance() which does RPC negociation | |
| 08:55:33 | bauzas | and then remove target_state if computes are older than rpc 6.2 | |
| 08:57:58 | noonedeadpunk | bauzas: is there any easy way to see reported prc version? becauce I guess in nova.services it's a different version rather then RPC. | |
| 08:58:00 | bauzas | noonedeadpunk: https://github.com/openstack/nova/commit/8c2e76598995f0d417653c60a63ea342baf4e880 is the commit that adds RPC 6.2 | |
| 08:58:24 | bauzas | noonedeadpunk: good question | |
| 08:58:58 | bauzas | I'd say internal RPC versions aren't meant to be exposed, but we provide aliases | |
| 08:59:20 | bauzas | https://github.com/openstack/nova/blob/stable/2023.1/nova/compute/rpcapi.py#L428 | |
| 08:59:45 | bauzas | zed computes are exposing 6.1, antelope ones are 6.2 | |
| 09:00:00 | noonedeadpunk | I guess I was more willing to check what computes/conductor reports as versions | |
| 09:00:08 | bauzas | if both your conductor *and* computes are 6.2, then try out to pin the rpc version | |
| 09:00:17 | bauzas | yeah, got it | |
| 09:00:28 | bauzas | the service version should tho give you that info | |
| 09:00:46 | noonedeadpunk | so if all is 66 - it should kinda work... | |
| 09:00:54 | bauzas | https://github.com/openstack/nova/commit/8c2e76598995f0d417653c60a63ea342baf4e880#diff-c0b6a5928be3ac40200a2078b084341bb9187a12b1f959ad862e0038c9029193 | |
| 09:01:09 | bauzas | 66 is the object version that matches RPC 6.2 | |
| 09:01:11 | noonedeadpunk | As iirc 66 is 2023.1 | |
| 09:01:14 | noonedeadpunk | yeah... | |
| 09:01:44 | bauzas | are you sure you restarted all your conductors after the upgrade ? :) | |
| 09:01:54 | bauzas | sorry for that silly question but I had to | |
| 09:02:02 | noonedeadpunk | They won't report 66 otherwise? | |
| 09:02:06 | bauzas | good call | |
| 09:02:17 | noonedeadpunk | I was actually trying to find a way to check what I could forgot to restart :) | |
| 09:02:31 | noonedeadpunk | But I'm checking now.... | |
| 09:02:51 | bauzas | I guess you can find the error in the conductor log, isn't it ? | |
| 09:03:02 | bauzas | if so, identify the conductor service in cause | |
| 09:10:04 | noonedeadpunk | Well, I've checked ps on all conductor hosts - all running from same venv and venv have same nova version installed... | |
| 09:10:21 | noonedeadpunk | And logs are clean as conductor doesn't obviously see any issues | |
| 09:10:40 | noonedeadpunk | let me check if code does contain these patches you've sent me | |
| 09:14:26 | noonedeadpunk | yeah, it's obviously there for all conductors... | |
| 09:21:10 | noonedeadpunk | Well, upgrade check throws `hw_machine_type unset` warning. Will try to sort it out - not sure if that's related though | |
| 09:23:51 | bauzas | no, this isn't | |
| 09:25:19 | bauzas | noonedeadpunk: since it's a venv, can you force to add some log in the conditional ? | |