| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-22 | |||
| 22:57:44 | sean-k-mooney | not directly. i work at redhat on the comptue team so i mainly work upstream but i also support customer when they report bugs downstream | |
| 22:58:13 | sean-k-mooney | so we have some large cloud deployment that i support indirely but im on the enginerring team rather then support | |
| 22:58:30 | lucidguy | I'm supporting a 600+ hypervisor Queens deployment. | |
| 22:59:05 | lucidguy | I'm betting my boss is going to ask me to upgrade to Train. Lord help me. | |
| 22:59:43 | sean-k-mooney | what do you use for your installer. charms? or do you use something like kolla ansible or osa? | |
| 23:00:15 | sean-k-mooney | queens to train is a bit leap but its doable | |
| 23:00:50 | lucidguy | We install everything from scratch, using custom ansible playbooks | |
| 23:01:02 | sean-k-mooney | oh ok | |
| 23:01:10 | lucidguy | Are we crazy? | |
| 23:01:13 | sean-k-mooney | ya that makes it more challanging in some ways | |
| 23:01:23 | sean-k-mooney | that depends | |
| 23:01:28 | lucidguy | But this way you trully understand how things work. | |
| 23:01:40 | sean-k-mooney | yep that is the advantage | |
| 23:01:52 | sean-k-mooney | i would suggest looking at the ansibel roles form OSA | |
| 23:02:04 | sean-k-mooney | osa being openstack ansibel | |
| 23:02:11 | sean-k-mooney | to so see if they are of use too you | |
| 23:02:34 | lucidguy | OpenStack ansible popular ehh? | |
| 23:02:42 | sean-k-mooney | yes | |
| 23:03:01 | sean-k-mooney | it is the install that vexhost use | |
| 23:03:10 | lucidguy | vexhost? | |
| 23:03:33 | sean-k-mooney | they are a large public cloud operator that donate ci resouce to the openstck comunity | |
| 23:03:45 | lucidguy | interesting | |
| 23:04:00 | sean-k-mooney | lucidguy: you shoudl reach out to mnaser | |
| 23:04:23 | sean-k-mooney | i think he is still the PTL of OSA but they have done a lot to make upgrade simpler | |
| 23:04:59 | lucidguy | Doesn't canonical support OpenStack? | |
| 23:05:19 | sean-k-mooney | yes they do, they have an installer base on opesntack charms deployed with juju | |
| 23:05:58 | sean-k-mooney | so if you pay for support with them that is the install they woudl use | |
| 23:05:59 | lucidguy | interesting | |
| 23:06:21 | lucidguy | I would think large organizations would go with our approach. Avoid wrappers. | |
| 23:06:28 | sean-k-mooney | redhat has an openstack distobution that build on top of the triplo project | |
| 23:06:43 | sean-k-mooney | you would be surprised | |
| 23:07:03 | sean-k-mooney | some do if they have the devepolment staff to mainatin it | |
| 23:07:25 | sean-k-mooney | but its more common to see large companies contibuie and addopt the comuntiy installer | |
| 23:07:37 | sean-k-mooney | *installers | |
| 23:08:41 | lucidguy | Just sounds like a pain to have to troubleshoot another layer of software. | |
| 23:10:28 | sean-k-mooney | well when that software is built by a group of operators that have really clouds in produciton you would be suprised at how helpful it can be | |
| 23:10:53 | sean-k-mooney | that said i agree if its a complex installer it can be a pain to debug | |
| 23:11:24 | sean-k-mooney | so i tend to prefer the simpler ones like devstack and kolla-ansible | |
| 23:11:37 | sean-k-mooney | just never run devstack in production... | |
| 23:14:59 | lucidguy | ugg stupid instance stuck in deleting state.. and virsh on hv not responding FML | |
| 23:15:44 | sean-k-mooney | that sounds like the host kernel might have hung | |
| 23:16:18 | lucidguy | this is the box I disabled nesting | |
| 23:16:19 | sean-k-mooney | by the way i hope if you are runing 1TB VM you are using Hugepages for the memory | |
| 23:16:36 | lucidguy | Whatever the default option is. | |
| 23:16:44 | sean-k-mooney | so no you are not | |
| 23:17:12 | sean-k-mooney | the reason i mentioned that is it can take a long time for qemu/kvm to release all the mapped memory when you kill it | |
| 23:17:16 | lucidguy | I was just reading those options in nova.conf | |
| 23:17:40 | sean-k-mooney | so if you are using the decault 4k memory then it might take a while to fully delete the vm | |
| 23:18:19 | lucidguy | Would I not see a load on the system anywhere? | |
| 23:19:37 | lucidguy | I don't even see a qemu process running | |
| 23:21:05 | sean-k-mooney | you would see that the memory has not been freed with free -m | |
| 23:21:24 | sean-k-mooney | but in that case if virsh is not working i would check to see if libvirt is still running | |
| 23:21:31 | lucidguy | Its free | |
| 23:21:50 | lucidguy | libvirtError: failed to connect to monitor socket: No such process | |
| 23:22:16 | sean-k-mooney | so that mean libvirt tried to connecct to qemu but it failed because it was not running | |
| 23:22:23 | lucidguy | oooo its gone. So it did take forever. | |
| 23:22:44 | lucidguy | Now you're going to make me research how Hugepages works. Thanks alot. heh | |
| 23:23:05 | sean-k-mooney | hehe ya they are non trival | |
| 23:23:24 | sean-k-mooney | but basically it invole preallcoating meory so that it can be handed out in larger chunks | |
| 23:23:35 | sean-k-mooney | normaly either 2MB or 1G instead of 4k | |
| 23:23:56 | sean-k-mooney | that signicatly impoves memory performance but it has other implciations | |
| 23:24:32 | lucidguy | Won't the mem be used less efficiently? | |
| 23:24:34 | sean-k-mooney | the main onces beign non memory over subsription and no live migation suppor until train | |
| 23:24:41 | sean-k-mooney | no | |
| 23:25:09 | sean-k-mooney | by using larger page allocation you actully optimeis adress translation | |
| 23:25:49 | sean-k-mooney | the translation lookaside buffer can only cache a limite number of page translations. if each page is bigger you can cache the adress to more memory | |
| 23:26:24 | sean-k-mooney | so it can give 30%+ performanc improment if you applciation is memory or latency sensitive | |
| 23:26:59 | lucidguy | Tried launching a new intance on that HV without nexting.. instance comes up paused and I can see that same error in libvirt .. :( | |
| 23:27:17 | lucidguy | Was worth a try. | |
| 23:27:24 | sean-k-mooney | ya sorry it did not work | |
| 23:28:23 | lucidguy | Was so excited about that. Installing a 5.0 kernel on a 16.04 box in production I don't think is wise | |
| 23:28:52 | sean-k-mooney | i run 5.4 on my home server based on ubuntu 18.04 | |
| 23:29:14 | sean-k-mooney | but its really something you need to evaluate for your own usecase | |
| 23:29:27 | lucidguy | Did you basically upgrade using this approach? https://www.tecmint.com/upgrade-kernel-in-ubuntu/ | |
| 23:30:20 | sean-k-mooney | no i was lazy and used ukuu the ubuntu kernel update utility | |
| 23:30:24 | sean-k-mooney | it automates that | |
| 23:30:37 | lucidguy | ohh, theres a cli approach. | |
| 23:30:37 | sean-k-mooney | but i am running the 5.4 mainline kernel form that ppa | |
| 23:31:07 | sean-k-mooney | https://www.omgubuntu.co.uk/2017/02/ukuu-easy-way-to-install-mainline-kernel-ubuntu | |
| 23:32:01 | sean-k-mooney | again you proably shoudl not do that in prodcutin but i basically un the latest upstream lts kernel which is 5.4 a the moment | |
| 23:32:17 | sean-k-mooney | 5.0 was the previous upstream long life kernel i think | |
| 23:33:14 | sean-k-mooney | oh the previous long term support was 4.19 | |
| 23:33:18 | lucidguy | Things to think about. Thanks again. Have to run. Wife is hating me. heh | |
| 23:33:31 | sean-k-mooney | https://www.kernel.org/category/releases.html | |
| 23:33:37 | sean-k-mooney | no worries o/ | |
| 23:34:50 | lucidguy | I guess I should start by updating my local repo and giving 4.19 a try. Sadly I'm not optimistic of the outcome. | |
| 23:35:11 | lucidguy | have a good one | |
| 23:55:21 | efried | jroll: we had talked vaguely about you being involved in the development of the CI, so I thought to involve you in that conversation :) | |
| #openstack-nova - 2020-01-23 | |||
| 08:31:30 | openstackgerrit | Merged openstack/nova master: Func test for failed and aborted live migration https://review.opendev.org/699065 | |
| 08:31:36 | openstackgerrit | Merged openstack/nova master: Remove unused code https://review.opendev.org/703408 | |
| 09:27:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove remaining Python 2.7-only dependencies https://review.opendev.org/698895 | |
| 09:33:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova.image.api' module https://review.opendev.org/702451 | |
| 09:33:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Update API reference guide https://review.opendev.org/703796 | |
| 09:33:55 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: nova-net: Remove unused nova-network objects https://review.opendev.org/697156 | |
| 11:21:15 | openstackgerrit | Merged openstack/nova master: Switch to hacking 2.x https://review.opendev.org/695733 | |
| 11:21:20 | openstackgerrit | Merged openstack/nova master: requirements: Limit hacking to one minor version https://review.opendev.org/703405 | |
| 11:32:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156 | |
| 11:32:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused parameters https://review.opendev.org/703974 | |
| 11:32:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unnecessary exception handling https://review.opendev.org/703973 | |
| 11:34:09 | stephenfin | dansmith: I'm doing possibly bad stuff with objects in https://review.opendev.org/697156. I _think_ it's okay but would appreciate your input when you're around | |