| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-15 | |||
| 18:34:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Store UEFI'ness of guest as attribute https://review.opendev.org/c/openstack/nova/+/775692 | |
| 18:34:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: libvirt: Stop checking non-host architectures for SEV https://review.opendev.org/c/openstack/nova/+/775693 | |
| 18:39:49 | kukacz | sean-k-mooney: I might retry the perf with xfs. thank you for the parameters. I don't target for top performance comparing lvm-raw-qcow. say 20% difference is perfectly ok if there is better implementation of qcow. just the huge drop (700k -> 30k) I'm seeing is too much already. I have to recheck the formatting/mount options. | |
| 18:41:52 | sean-k-mooney | raw shoudl basically give you the same performace as lvm with a small amount of overhead for ext4/xfs | |
| 18:41:59 | sean-k-mooney | but you can resize ectra | |
| 18:45:03 | kukacz | yes, with raw I'm getting good numbers. resize did not work for me today though, perhaps I hit a bug - resizing an instance with raw disk in queens created an unbootable qcow image | |
| 18:45:33 | sean-k-mooney | did you have the for raw disk set | |
| 18:45:53 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.force_raw_images | |
| 18:46:47 | kukacz | yes, I have that set to True | |
| 18:47:22 | sean-k-mooney | and https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.use_cow_images set to false | |
| 18:49:23 | kukacz | no, I have use_cow_images unset, which probably renders true | |
| 18:49:45 | sean-k-mooney | true is the default | |
| 18:50:12 | aarents | kukacz: seems odd, we are heavy user of raw and resize are fine, we have use_cow_images to false as sean suggest in newton and stein | |
| 18:51:20 | kukacz | aarents: interesting. I'll try that configuration | |
| 18:51:59 | kukacz | though I did not have other issues with raw image deployments and backing files. only issue was this resize | |
| 18:52:33 | sean-k-mooney | so we removed teh raw backend a while ago and folded it into the flat backend https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L1286 | |
| 18:53:06 | sean-k-mooney | and it gets is resize form ate dynamically https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L593-L595 | |
| 18:54:31 | sean-k-mooney | i think this is what decideds https://github.com/openstack/nova/blob/0503bb14409a1a78a82fd8e6f931259fde753df9/nova/virt/libvirt/imagebackend.py#L537-L545 | |
| 18:55:30 | sean-k-mooney | kukacz: perhaps qemu-image info was retruning something other then raw? | |
| 18:57:58 | kukacz | sean-k-mooney: yes, file tool identified the disk as "QEMU QCOW Image (v3)" | |
| 18:58:46 | sean-k-mooney | before or after the resize | |
| 18:59:16 | kukacz | now I have retried with `use_cow_images = false` and that works | |
| 18:59:55 | sean-k-mooney | did you explictly set teh images_type to raw? | |
| 18:59:57 | kukacz | sean-k-mooney: that was after resize. original disk is identified as "DOS/MBR boot sector" = raw | |
| 19:00:35 | kukacz | sean-k-mooney: yes, explicitely `images_type=raw` | |
| 19:00:37 | sean-k-mooney | i suspect tha twehe we merged raw into flat the behavior changed slightly and now you neeed to set use_cow_iamges=false | |
| 19:00:56 | sean-k-mooney | ya its not an alais for flat which wallows qcow images | |
| 19:01:28 | sean-k-mooney | it just does not crate a backing files when it uses qcow image like the normal qcow one those | |
| 19:07:29 | sean-k-mooney | stephenfin: bauzas so i figured out my fixture issue. i was expecting to use the libvirtneutron fixture but i was using the base one. | |
| 19:07:44 | sean-k-mooney | once i fixed that my func tests started passing. | |
| 19:12:14 | openstackgerrit | sean mooney proposed openstack/nova master: support per port numa policies with sriov https://review.opendev.org/c/openstack/nova/+/773792 | |
| 19:14:21 | kukacz | sean-k-mooney, aarents: thank you! the resize issue is resolved. I did couple more tests and seems ok now. | |
| 19:14:35 | sean-k-mooney | cool | |
| 19:15:17 | sean-k-mooney | raw should be very very close to lvm so if that works for you then it will give you the advanate of better testing and more features too | |
| 19:19:57 | kukacz | sean-k-mooney: hmm, that sounds good. so lvm is the least tested option compared to qcow/raw? not being used in production deployments as often? | |
| 19:25:54 | sean-k-mooney | that is my understnading yes most peopel that dont use qcow for local sotrage use raw then lvm is less common after that | |
| 19:27:00 | kukacz | there's another factor I've registered: with LVM I could expand capacity instantly just by adding a PV into VG. with qcow2/raw, I was using mdadm level 0, which needs to redistribute blocks while adding a disk. that was such a performance impacting and lengthy operation, that I could not be doing this online - with production workload running. | |
| 19:28:50 | sean-k-mooney | well you can just put the raw files on an lvm volumn | |
| 19:28:51 | kukacz | but I could move the LVM to OS level here, format, mount to /var/lib/nova and keep the disk management benefits perhaps, while utilitizing better integrated qcow or raw backends | |
| 19:28:56 | sean-k-mooney | and expand it the same way | |
| 19:30:06 | kukacz | sean-k-mooney: exactly :-) | |
| 19:30:07 | sean-k-mooney | using raw/qcow does not percluded you mounting the nova state dir on an lvm volumn seperatly | |
| 19:33:15 | kukacz | sean-k-mooney: you are right, it does not. I have just defined myself different test scenarios originally, not including this one | |
| 19:34:47 | sean-k-mooney | my home openstack has my nova directory in an lvm vg on a raid 6 disk array with an optane ssd as a cache via bcache | |
| 19:35:08 | kukacz | with the current knowledge, it seems the best option. supposing I can get to slightly comparable performance | |
| 19:36:29 | sean-k-mooney | in your case you dont need bcach since its all ssd but for me its nice to hide some of the latency of HDDs | |
| 19:36:50 | sean-k-mooney | anyway o/ | |
| 19:40:11 | kukacz | of course. I'm also trying to keep the configuration as simple as possible. avoiding any extra layers. that's why I'm so enthusiastic of the idea of direct LVM usage | |
| 19:42:12 | kukacz | if I put filesystem (/var/lib/nova/instances) on top of LVM, it will not partially-tolerate disk failure anymore. that could only be achieved with direct Nova-LVM integration | |
| 19:43:35 | kukacz | seems that I'll need to accept a compromise | |
| #openstack-nova - 2021-02-16 | |||
| 09:03:08 | lyarwood | Morning | |
| 09:03:42 | xarlos | o/ | |
| 09:13:40 | lyarwood | gibi / stephenfin ; thanks again for the reviews yesterday, just going over the initial changes now, if you have time this morning I'd appreciate any reviews on the latter changes. | |
| 09:25:05 | stephenfin | ack | |
| 09:26:15 | openstackgerrit | Belmiro Moreira proposed openstack/nova master: Change API unexpected exception message https://review.opendev.org/c/openstack/nova/+/628061 | |
| 09:37:04 | belmoreira | Hello, just had another user confused with the API error: 'Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.' and I remember that we already discussed it. I would like to revive it. Do you still think that a config option is the way to go? Because I can't see how this specific error message is good for any cloud provider. | |
| 09:37:08 | belmoreira | https://review.opendev.org/c/openstack/nova/+/628061 | |
| 09:37:37 | belmoreira | let me know your comments. I would like to work on it. | |
| 09:41:39 | lyarwood | belmoreira: +1 to making it configurable somehow | |
| 09:42:02 | lyarwood | belmoreira: we already have vendor specific configurables ./me checks | |
| 09:45:26 | lyarwood | belmoreira: https://github.com/openstack/nova/blob/master/nova/version.py is what I was thinking of | |
| 09:45:40 | lyarwood | belmoreira: adding some form of bug reporting URL there that defaults to http://bugs.launchpad.net/nova/ would be nice | |
| 09:46:26 | lyarwood | belmoreira: and you could overwrite that in your deployment to point at your support portal etc | |
| 09:47:46 | lyarwood | https://github.com/rdo-packages/nova-distgit/blob/73a5163768f8d093867e1f7d9ea1129e4a0a839a/openstack-nova.spec#L525-L531 is how we do this downstream in RDO and OSP | |
| 09:48:07 | lyarwood | well, overwrite the vendor, product and package | |
| 09:48:24 | lyarwood | and tbh we could also use a bug_tracker configurable that points to our support portal there | |
| 09:48:37 | belmoreira | lyarwood thanks, let me check | |
| 09:54:12 | lyarwood | gah these gerrit comment threads are confusing | |
| 09:58:29 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/769548/10/nova/tests/unit/virt/test_machine_type_utils.py - stupid question, do you mean drop the empty line? | |
| 09:58:55 | stephenfin | oh, yeah | |
| 09:59:04 | stephenfin | we haven't switched to unittest.mock yet | |
| 09:59:15 | lyarwood | right I was about to say | |
| 09:59:22 | lyarwood | I thought I had missed something with unittest.mock | |
| 09:59:29 | lyarwood | cool | |
| 10:54:29 | gibi | lyarwood: ack, I'm slowly reviewing the rest of the machine_type series but there is a lot of distraction around me at the moment | |
| 10:55:10 | lyarwood | gibi: np at all, take your time I appreciate it all the same :) | |
| 10:55:25 | lyarwood | >< | |
| 10:55:34 | lyarwood | I'd be made if it wasn't so cute | |
| 10:55:36 | lyarwood | mad* | |
| 10:55:52 | stephenfin | bauzas: Want to take a quick look at https://review.opendev.org/c/openstack/os-traits/+/759878 so we can cut an os-traits release? | |
| 10:56:40 | bauzas | stephenfin: currently fighting with the odds of _IntegratedTestBase but sure can look :) | |
| 10:57:07 | gibi | lyarwood: you won | |
| 10:57:20 | stephenfin | gibi: I've proposed an os-traits 2.5.0 release https://review.opendev.org/c/openstack/releases/+/775804 and think we could merge it without https://review.opendev.org/c/openstack/os-traits/+/759878 | |
| 10:57:24 | stephenfin | seeing as releases are cheap | |
| 10:57:25 | bauzas | fwiw we're 99% on par with ProviderUsageBaseTestCase but I hit the 1% left :) | |
| 10:58:03 | bauzas | yeah we can release it | |
| 10:58:38 | bauzas | and here I'm ok with playing the same team card | |
| 10:58:44 | bauzas | it's just a trait | |
| 10:59:09 | stephenfin | thanks | |
| 10:59:17 | gibi | hm so https://review.opendev.org/c/openstack/os-traits/+/759878 is now approved | |
| 10:59:23 | gibi | then I guess there will be a new release patch | |
| 10:59:41 | stephenfin | Yeah, might as well | |
| 11:00:01 | gibi | (I would have approved that os-trait patch by myself but bauzas was faster) | |
| 11:00:14 | gibi | stephenfin: I think the os-trait merge should not take long time | |
| 11:00:18 | bauzas | gibi: I added a comment about it | |
| 11:00:25 | bauzas | about the same company one | |
| 11:00:50 | bauzas | gibi: agreed about releasing it quickly | |
| 11:01:06 | gibi | bauzas: no worries about the same company rule | |
| 11:02:36 | gibi | stephenfin: is it OK for you to wait for this patch to merge and then release the os-traits? | |
| 11:02:45 | stephenfin | yup, wfm | |