| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-15 | |||
| 11:53:12 | sean-k-mooney | by the way why i said scsi isnt really an option is windows does not have teh virtio-scsi drivers by default | |
| 11:53:19 | sean-k-mooney | so we could not change the default to scsi | |
| 11:53:33 | sean-k-mooney | we chose sata becasue it shoudl work on all operating systmes | |
| 11:53:42 | lyarwood | is scsi == virtio-scsi? | |
| 11:53:51 | lyarwood | you have to set the model to virtio-scsi for that to be the case right? | |
| 11:54:01 | sean-k-mooney | yes you do | |
| 11:54:12 | sean-k-mooney | the other scsi contoller are all quite old | |
| 11:54:20 | sean-k-mooney | im not sure we would want to use any of those by default | |
| 11:54:31 | lyarwood | well at least Windows would support it then | |
| 11:54:34 | lyarwood | but yeah | |
| 11:54:58 | sean-k-mooney | im not sure if it would it might | |
| 11:55:07 | sean-k-mooney | using scsi works by the way for cirros | |
| 11:55:12 | sean-k-mooney | show up as normal | |
| 11:56:03 | lyarwood | not for me as the config cdrom drive | |
| 11:56:10 | lyarwood | so I think there's something still missing there | |
| 11:56:17 | lyarwood | I found https://bugs.launchpad.net/cirros/+bug/1715009 from a while ago | |
| 11:56:19 | openstack | Launchpad bug 1715009 in CirrOS "Missing modules for scsi cdrom config_drive in ppc64le" [Medium,Fix committed] | |
| 11:56:34 | lyarwood | so I wonder if there's additional things we need on x86_64 | |
| 11:57:02 | sean-k-mooney | so looking at /lib/modules/... | |
| 11:57:14 | sean-k-mooney | cirros hs sfcsi and virtio drivers | |
| 11:57:17 | sean-k-mooney | but not sata | |
| 11:57:52 | sean-k-mooney | it only has the virtio scsi drivers by the way | |
| 11:58:16 | sean-k-mooney | im guessing the init ramfs has the sata drivers | |
| 11:58:45 | sean-k-mooney | or it fell back to an in kernel driver | |
| 11:58:51 | sean-k-mooney | rahter then a module | |
| 12:02:37 | sean-k-mooney | lyarwood: by the way we have a bug with bfv | |
| 12:02:58 | sean-k-mooney | lyarwood: shocking i know but bfv does not select the correct bus when you use sata | |
| 12:03:04 | sean-k-mooney | it will use scsi instead | |
| 12:03:13 | sean-k-mooney | i fixed this years ago for non bfv | |
| 12:03:27 | sean-k-mooney | but it looks like it back or was never fixed for bfv | |
| 12:03:46 | kashyap | Sorry for my choppiness here. Struggling with something today. | |
| 12:08:13 | openstackgerrit | Hemanth N proposed openstack/nova stable/rocky: Update pci stat pools based on PCI device changes https://review.opendev.org/c/openstack/nova/+/761824 | |
| 12:26:48 | lyarwood | sean-k-mooney: sorry had to go afk, with hw_disk_bus=sata? | |
| 12:30:13 | sean-k-mooney | lyarwood: ya if you hw_disk_bus=sata with bfv on ussuri then you get scsi | |
| 13:02:12 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Record the machine_type of instances in system_metadata https://review.opendev.org/c/openstack/nova/+/767533 | |
| 13:02:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type update command https://review.opendev.org/c/openstack/nova/+/774896 | |
| 13:02:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type get command https://review.opendev.org/c/openstack/nova/+/769548 | |
| 13:02:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-status: Add hw_machine_type check for libvirt instances https://review.opendev.org/c/openstack/nova/+/770643 | |
| 13:02:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-manage: Add machine_type list_unset command https://review.opendev.org/c/openstack/nova/+/774897 | |
| 13:02:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: docs: Add admin docs for configuring and updating machine types https://review.opendev.org/c/openstack/nova/+/774899 | |
| 13:02:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add a config update workflow test for [libvirt]hw_machine_type https://review.opendev.org/c/openstack/nova/+/774898 | |
| 13:57:15 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/c/openstack/nova/+/708701 | |
| 15:36:43 | lyarwood | stephenfin: thanks for the review, would you mind if I hold off on that respin until the rest of the series has been looked at? ./me doesn't want to hammer CI | |
| 15:36:57 | stephenfin | yup, fine by me. Working through it atm | |
| 15:37:51 | lyarwood | stephenfin: excellent thanks | |
| 15:38:08 | openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: DO NOT REVIEW: Test OVN devstack module https://review.opendev.org/c/openstack/nova/+/748226 | |
| 15:53:19 | kukacz | hi. having queens, nova disk with images_type=raw, when doing server resize I'm ending up with unbootable instance. I noticed that new (resized up) boot disk is being created as qcow, not raw. Is this a bug or am I rather missing some configuration detail? | |
| 15:57:39 | lyarwood | kukacz: that sounds like a bug, can you open one and share some logs of the resize? | |
| 15:58:59 | lyarwood | https://launchpad.net/nova/+bug for the bug btw if you didn't know | |
| 16:06:22 | kukacz | lyarwood: sure, thanks, I'll collect logs+details into the bug. I'll need to switch the server to different configuration (lvm backend) for production, most probably not being able to collect more details or tests into the bug when required for some weeks. is it reasonable to file the bug despite such constraint? | |
| 16:08:02 | lyarwood | kukacz: yup I might try to reproduce this tomorrow anyway | |
| 16:09:38 | kukacz | lyarwood: perfect! I'll do that then | |
| 16:10:53 | kukacz | btw. what is the current status of LVM backen resize support in Nova? I did a similar test with it and was (correctly) refused, stating it in logs that it's not supported. has that changed in newer releases? | |
| 16:11:20 | lyarwood | kukacz: I don't think so but let me check | |
| 16:19:30 | lyarwood | yeah I don't think it is sorry | |
| 16:34:37 | kukacz | no worries, thanks for checking that! | |
| 16:40:28 | kukacz | lyarwood: is the LVM backend in a good shape otherwise? I am planning to deploy "local disk" compute nodes utilizing LVM as image type after doing own evaluation, comparing it to qcow2/raw options. | |
| 16:41:33 | lyarwood | kukacz: It doesn't get as much love as the default qcow2 backend and isn't something we support downstream as a result | |
| 16:53:38 | kukacz | lyarwood: hmm, I see. I've found LVM superior both in performance characteristics (qcow2 on ext4 was badly stealing most of NVMe random pattern iops) and availability-operations capabililites though | |
| 16:55:06 | kukacz | lyarwood: when writing "we support ..." you meant a particular vendor or Nova project? | |
| 16:55:35 | lyarwood | kukacz: Vendor sorry, Red Hat. | |
| 16:56:00 | kukacz | I am considering putting this into production, therefore I am so curious :-) | |
| 16:56:41 | kukacz | ok, it might even mean both in that case ;-) | |
| 16:59:53 | kukacz | I might still be missing something. especially I wonder if that performance hit on qcow2 (on default ext4 on 1-2 disk mdraid level 0) is common or there's an factor I've ignored so far | |
| 17:59:52 | sean-k-mooney | kukacz: qcow2 tends not to have a large perfromnce hit if you preallocate the file by the way | |
| 18:00:11 | sean-k-mooney | at least not on an xfs host filesystem | |
| 18:00:40 | sean-k-mooney | im not sure you really rant to run raid 0 | |
| 18:01:11 | sean-k-mooney | raid 10 maybe but unless you hate your customers data raid 0 is proably not what you want to use in production | |
| 18:05:59 | kukacz | sean-k-mooney: I had the preallocation enabled while testing. the performance was good in sequential operations (>3300MiB/s), not in random though. in random, the 750k IOPS seen in lvm or raw are dropping to around 30k. tested on precreated 80GiB large fio file, running concurrently 6 instances | |
| 18:06:32 | kukacz | I was using ext4 without extra parameters except -m0. same as with raw, though | |
| 18:07:00 | sean-k-mooney | kukacz: try turing off the atime? i think | |
| 18:07:57 | kukacz | hmm, I believe fio keeps the single testfile opened for all the time of test. but I might try that anyway | |
| 18:08:02 | sean-k-mooney | for xfs i do defaults,noatime,nodiratime,logbufs=8,logbsize=256k,largeio but you will proably want something similar for ext4 | |
| 18:08:25 | sean-k-mooney | oh i have /var/lib/docker ext4 defaults,noatime 0 0 | |
| 18:09:06 | sean-k-mooney | kukacz: historically | |
| 18:09:30 | sean-k-mooney | kukacz: lvm had better io performacne but it took longer to deploy vms and used more space | |
| 18:10:02 | sean-k-mooney | basically becasue even when using thing providioning it need to write all the data for the base iamge every time a vm boots | |
| 18:10:31 | sean-k-mooney | we dont do any kind of snapshoting in the lvm driver to merged multiple thin provisioned images | |
| 18:10:55 | sean-k-mooney | for the qcow backedn we share a backing disk and jsut create a thin snapshot | |
| 18:11:21 | kukacz | I was not enabling thin provisioning with LVM, if you mean that nova.conf parameter | |
| 18:11:28 | sean-k-mooney | so if you have long running vms lvm will likely preferm betere even if its less maintianed | |
| 18:12:06 | sean-k-mooney | if you have short lived vms the qcow is proably going to be better. | |
| 18:12:09 | kukacz | I wanted the extents to be preallocate in a predictable way - priorizing volumes to be placed on same disk, if it fits by size | |
| 18:14:38 | kukacz | hmm, the provisioning time was nothing I have even noticed with LVM, but I was using quite smallish but typical ubuntu 20 image (around 5GB) | |
| 18:15:17 | sean-k-mooney | are you on flash | |
| 18:15:32 | sean-k-mooney | if you have ssd then you wont notice it too much | |
| 18:15:44 | sean-k-mooney | if you have HDD and you have multiple vms starting you will | |
| 18:15:52 | kukacz | raid0 was intentional here - focus on performance. if customer is using local disk, they assume that the single instance might be lost anyway. targetting cloud-native replicated applications | |
| 18:16:02 | kukacz | yes, flash, NVMe SSDs | |
| 18:16:27 | sean-k-mooney | kukacz: well jus tmake sure they are aware of that | |
| 18:16:49 | sean-k-mooney | i assume you will be providing them with a ha cinder solution? ceph? | |
| 18:17:03 | sean-k-mooney | so tha they can store all there main data elsewhere | |
| 18:17:04 | kukacz | of course, they're having other options for more traditional network storage | |
| 18:17:30 | kukacz | yes, ceph as cinder backend | |
| 18:17:55 | sean-k-mooney | so local lvm is for low latency local storage | |
| 18:18:12 | sean-k-mooney | for root files system/scratch space | |
| 18:18:39 | sean-k-mooney | it used to work resonbably well for that | |
| 18:18:57 | sean-k-mooney | i is tempthing to just use ceph for everything if you have it | |
| 18:19:17 | sean-k-mooney | possible deploying osd on the compute nodes too | |
| 18:19:49 | sean-k-mooney | if you find the perfomce of lvm is not worth it the rbd driver would be worht considering | |