| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-15 | |||
| 18:17:30 | kukacz | yes, ceph as cinder backend | |
| 18:17:55 | sean-k-mooney | so local lvm is for low latency local storage | |
| 18:18:12 | sean-k-mooney | for root files system/scratch space | |
| 18:18:39 | sean-k-mooney | it used to work resonbably well for that | |
| 18:18:57 | sean-k-mooney | i is tempthing to just use ceph for everything if you have it | |
| 18:19:17 | sean-k-mooney | possible deploying osd on the compute nodes too | |
| 18:19:49 | sean-k-mooney | if you find the perfomce of lvm is not worth it the rbd driver would be worht considering | |
| 18:20:30 | kukacz | we've had used rbd driver so far as nova backend | |
| 18:21:52 | kukacz | still, ceph is a shared network storage. if it has an issue, everyone in the cloud has an issue. very binding for the operators | |
| 18:22:46 | kukacz | while there's an amount of workload which does not need the features of shared storage. also, they replicate blocks by themselves already, and storing those on ceph, which creates additional copies is counter-productive | |
| 18:24:51 | kukacz | so the idea is to offer a storage layer for these workloads. it would get much better latencies, much higher iops, not being exposed to risk of central storage issue | |
| 18:26:15 | sean-k-mooney | it a valid concern and ya that is what many doo | |
| 18:27:17 | sean-k-mooney | if you can live with it being shared storage you can also modify the crush map to create an ssd only pool with no replication and expsoe that as a seperate cinder volume type | |
| 18:27:42 | sean-k-mooney | anyway if you want best performce then you shoudl use eivehr lvm or raw | |
| 18:28:28 | sean-k-mooney | qcow will be slower the both the other but it has some advnatges if customer will use thing like snapshopt | |
| 18:29:34 | kukacz | so far, I've been quite excited with the LVM experience in this usage. only issue not being able to resize | |
| 18:31:43 | kukacz | I acn even lose a drive from linear VG, only dropping the volumes stored on that particular drive. the rest of workload on same VG keeps running. which is also an advantage compared to raid-0 | |
| 18:31:49 | kukacz | I can ... | |
| 18:34:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Auto-detect UEFI, secure boot support https://review.opendev.org/c/openstack/nova/+/682627 | |
| 18:34:05 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/c/openstack/nova/+/673790 | |
| 18:34:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: libvirt: Methods to handle request for Secure Boot & non-Q35 machine types https://review.opendev.org/c/openstack/nova/+/682628 | |
| 18:34:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add missing hints https://review.opendev.org/c/openstack/nova/+/775688 | |
| 18:34:07 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Add secure boot loader config support https://review.opendev.org/c/openstack/nova/+/775687 | |
| 18:34:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Report secure boot support to scheduler https://review.opendev.org/c/openstack/nova/+/775690 | |
| 18:34:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Stop passing around virt_type, caps https://review.opendev.org/c/openstack/nova/+/775689 | |
| 18:34:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: libvirt: Switch to libvirt's firmware auto-selection https://review.opendev.org/c/openstack/nova/+/775691 | |
| 18:34:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: libvirt: Stop checking non-host architectures for SEV https://review.opendev.org/c/openstack/nova/+/775693 | |
| 18:34:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Store UEFI'ness of guest as attribute https://review.opendev.org/c/openstack/nova/+/775692 | |
| 18:39:49 | kukacz | sean-k-mooney: I might retry the perf with xfs. thank you for the parameters. I don't target for top performance comparing lvm-raw-qcow. say 20% difference is perfectly ok if there is better implementation of qcow. just the huge drop (700k -> 30k) I'm seeing is too much already. I have to recheck the formatting/mount options. | |
| 18:41:52 | sean-k-mooney | raw shoudl basically give you the same performace as lvm with a small amount of overhead for ext4/xfs | |
| 18:41:59 | sean-k-mooney | but you can resize ectra | |
| 18:45:03 | kukacz | yes, with raw I'm getting good numbers. resize did not work for me today though, perhaps I hit a bug - resizing an instance with raw disk in queens created an unbootable qcow image | |
| 18:45:33 | sean-k-mooney | did you have the for raw disk set | |
| 18:45:53 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.force_raw_images | |
| 18:46:47 | kukacz | yes, I have that set to True | |
| 18:47:22 | sean-k-mooney | and https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.use_cow_images set to false | |
| 18:49:23 | kukacz | no, I have use_cow_images unset, which probably renders true | |
| 18:49:45 | sean-k-mooney | true is the default | |
| 18:50:12 | aarents | kukacz: seems odd, we are heavy user of raw and resize are fine, we have use_cow_images to false as sean suggest in newton and stein | |
| 18:51:20 | kukacz | aarents: interesting. I'll try that configuration | |
| 18:51:59 | kukacz | though I did not have other issues with raw image deployments and backing files. only issue was this resize | |
| 18:52:33 | sean-k-mooney | so we removed teh raw backend a while ago and folded it into the flat backend https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L1286 | |
| 18:53:06 | sean-k-mooney | and it gets is resize form ate dynamically https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L593-L595 | |
| 18:54:31 | sean-k-mooney | i think this is what decideds https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L537-L545 | |
| 18:55:30 | sean-k-mooney | kukacz: perhaps qemu-image info was retruning something other then raw? | |
| 18:57:58 | kukacz | sean-k-mooney: yes, file tool identified the disk as "QEMU QCOW Image (v3)" | |
| 18:58:46 | sean-k-mooney | before or after the resize | |
| 18:59:16 | kukacz | now I have retried with `use_cow_images = false` and that works | |
| 18:59:55 | sean-k-mooney | did you explictly set teh images_type to raw? | |
| 18:59:57 | kukacz | sean-k-mooney: that was after resize. original disk is identified as "DOS/MBR boot sector" = raw | |
| 19:00:35 | kukacz | sean-k-mooney: yes, explicitely `images_type=raw` | |
| 19:00:37 | sean-k-mooney | i suspect tha twehe we merged raw into flat the behavior changed slightly and now you neeed to set use_cow_iamges=false | |
| 19:00:56 | sean-k-mooney | ya its not an alais for flat which wallows qcow images | |
| 19:01:28 | sean-k-mooney | it just does not crate a backing files when it uses qcow image like the normal qcow one those | |
| 19:07:29 | sean-k-mooney | stephenfin: bauzas so i figured out my fixture issue. i was expecting to use the libvirtneutron fixture but i was using the base one. | |
| 19:07:44 | sean-k-mooney | once i fixed that my func tests started passing. | |
| 19:12:14 | openstackgerrit | sean mooney proposed openstack/nova master: support per port numa policies with sriov https://review.opendev.org/c/openstack/nova/+/773792 | |
| 19:14:21 | kukacz | sean-k-mooney, aarents: thank you! the resize issue is resolved. I did couple more tests and seems ok now. | |
| 19:14:35 | sean-k-mooney | cool | |
| 19:15:17 | sean-k-mooney | raw should be very very close to lvm so if that works for you then it will give you the advanate of better testing and more features too | |
| 19:19:57 | kukacz | sean-k-mooney: hmm, that sounds good. so lvm is the least tested option compared to qcow/raw? not being used in production deployments as often? | |
| 19:25:54 | sean-k-mooney | that is my understnading yes most peopel that dont use qcow for local sotrage use raw then lvm is less common after that | |
| 19:27:00 | kukacz | there's another factor I've registered: with LVM I could expand capacity instantly just by adding a PV into VG. with qcow2/raw, I was using mdadm level 0, which needs to redistribute blocks while adding a disk. that was such a performance impacting and lengthy operation, that I could not be doing this online - with production workload running. | |
| 19:28:50 | sean-k-mooney | well you can just put the raw files on an lvm volumn | |
| 19:28:51 | kukacz | but I could move the LVM to OS level here, format, mount to /var/lib/nova and keep the disk management benefits perhaps, while utilitizing better integrated qcow or raw backends | |
| 19:28:56 | sean-k-mooney | and expand it the same way | |
| 19:30:06 | kukacz | sean-k-mooney: exactly :-) | |
| 19:30:07 | sean-k-mooney | using raw/qcow does not percluded you mounting the nova state dir on an lvm volumn seperatly | |
| 19:33:15 | kukacz | sean-k-mooney: you are right, it does not. I have just defined myself different test scenarios originally, not including this one | |
| 19:34:47 | sean-k-mooney | my home openstack has my nova directory in an lvm vg on a raid 6 disk array with an optane ssd as a cache via bcache | |
| 19:35:08 | kukacz | with the current knowledge, it seems the best option. supposing I can get to slightly comparable performance | |
| 19:36:29 | sean-k-mooney | in your case you dont need bcach since its all ssd but for me its nice to hide some of the latency of HDDs | |
| 19:36:50 | sean-k-mooney | anyway o/ | |
| 19:40:11 | kukacz | of course. I'm also trying to keep the configuration as simple as possible. avoiding any extra layers. that's why I'm so enthusiastic of the idea of direct LVM usage | |
| 19:42:12 | kukacz | if I put filesystem (/var/lib/nova/instances) on top of LVM, it will not partially-tolerate disk failure anymore. that could only be achieved with direct Nova-LVM integration | |
| 19:43:35 | kukacz | seems that I'll need to accept a compromise | |
| #openstack-nova - 2021-02-16 | |||
| 09:03:08 | lyarwood | Morning | |
| 09:03:42 | xarlos | o/ | |
| 09:13:40 | lyarwood | gibi / stephenfin ; thanks again for the reviews yesterday, just going over the initial changes now, if you have time this morning I'd appreciate any reviews on the latter changes. | |
| 09:25:05 | stephenfin | ack | |
| 09:26:15 | openstackgerrit | Belmiro Moreira proposed openstack/nova master: Change API unexpected exception message https://review.opendev.org/c/openstack/nova/+/628061 | |
| 09:37:04 | belmoreira | Hello, just had another user confused with the API error: 'Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.' and I remember that we already discussed it. I would like to revive it. Do you still think that a config option is the way to go? Because I can't see how this specific error message is good for any cloud provider. | |
| 09:37:08 | belmoreira | https://review.opendev.org/c/openstack/nova/+/628061 | |
| 09:37:37 | belmoreira | let me know your comments. I would like to work on it. | |
| 09:41:39 | lyarwood | belmoreira: +1 to making it configurable somehow | |
| 09:42:02 | lyarwood | belmoreira: we already have vendor specific configurables ./me checks | |
| 09:45:26 | lyarwood | belmoreira: https://github.com/openstack/nova/blob/master/nova/version.py is what I was thinking of | |
| 09:45:40 | lyarwood | belmoreira: adding some form of bug reporting URL there that defaults to http://bugs.launchpad.net/nova/ would be nice | |
| 09:46:26 | lyarwood | belmoreira: and you could overwrite that in your deployment to point at your support portal etc | |
| 09:47:46 | lyarwood | https://github.com/rdo-packages/nova-distgit/blob/73a5163768f8d093867e1f7d9ea1129e4a0a839a/openstack-nova.spec#L525-L531 is how we do this downstream in RDO and OSP | |
| 09:48:07 | lyarwood | well, overwrite the vendor, product and package | |
| 09:48:24 | lyarwood | and tbh we could also use a bug_tracker configurable that points to our support portal there | |
| 09:48:37 | belmoreira | lyarwood thanks, let me check | |
| 09:54:12 | lyarwood | gah these gerrit comment threads are confusing | |
| 09:58:29 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/769548/10/nova/tests/unit/virt/test_machine_type_utils.py - stupid question, do you mean drop the empty line? | |
| 09:58:55 | stephenfin | oh, yeah | |
| 09:59:04 | stephenfin | we haven't switched to unittest.mock yet | |
| 09:59:15 | lyarwood | right I was about to say | |
| 09:59:22 | lyarwood | I thought I had missed something with unittest.mock | |
| 09:59:29 | lyarwood | cool | |