| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-08 | |||
| 12:31:22 | sean-k-mooney | kashyap: i also know that we are getting asked to consider added memory balloan managemnt downstream | |
| 12:32:58 | frickler | sean-k-mooney: ah, I see, you'd need to use my patch https://review.opendev.org/704831 , apart from the tempest failures it has been running pretty well for me for some months now | |
| 12:34:01 | sean-k-mooney | frickler: cool i was planning on swapping to it for the numa work im starting. the vms i have created are already on it but i had not got as far as stacking yey | |
| 12:34:49 | sean-k-mooney | so if i use that devstack patch it should be good | |
| 12:34:51 | frickler | sean-k-mooney: the ceph thing seems to have been fixed with https://review.opendev.org/725812 , not sure I've tested with that patch in tempest | |
| 12:35:13 | frickler | sean-k-mooney: yes, that would be great, let me know if you find any further issues | |
| 12:36:01 | sean-k-mooney | ill set up the multi node env and before i start modifying code ill try and do a few volume tests i assuem you are hitting the issue locally too? | |
| 12:36:08 | sean-k-mooney | e.g. its not just in the ci | |
| 12:36:43 | frickler | sean-k-mooney: yes, although locally I didn't see it 100%, might be a bit load/memory pressure related | |
| 12:37:21 | sean-k-mooney | hehe its funny but i realised the other day i finally got https://opendev.org/opentack into my muscel memory instead of defaulting to github | |
| 12:38:18 | sean-k-mooney | frickler: so while your here | |
| 12:38:21 | sean-k-mooney | ubuntu@numa-1:/opt/repos/devstack$ python3 -m pip install --user git-review | |
| 12:38:23 | sean-k-mooney | /usr/bin/python3: No module named pip | |
| 12:38:35 | sean-k-mooney | am i imagining things or is pip not part of the standard libary now | |
| 12:38:41 | sean-k-mooney | i know how to fix this | |
| 12:38:55 | sean-k-mooney | but im just wondering why on 3.8 i have to install a seperate package | |
| 12:39:05 | sean-k-mooney | i tought it was ment to be always present | |
| 12:39:47 | frickler | yeah, they did strange things there, same for venv, no idea why. for git-review, I just use the pkg from universe, not pip | |
| 12:40:26 | sean-k-mooney | i have had issue on centos7 with it being old and busted | |
| 12:40:37 | sean-k-mooney | so i default to pip for it | |
| 12:41:04 | sean-k-mooney | mainly to have a consitent behavior across distos to be honest | |
| 12:43:30 | frickler | I also had to create a symlink for pip3.8 since the distro pkg only provides pip3, but devstack wants it version specific | |
| 12:43:43 | sean-k-mooney | frickler: i think they are treating pip and venv as dev tools | |
| 12:44:27 | sean-k-mooney | frickler: we proably should modify devstack to not use the pip command but instaead do it the python3 -m pip way | |
| 12:45:05 | sean-k-mooney | it will avoid that issue but also i think that is the new recomendation to ensure it installs where you think it will | |
| 12:45:34 | sean-k-mooney | obviously python3 can be python3.8 or the version specifc one too | |
| 13:06:13 | sean-k-mooney | gibi: bauzas if we set map_az_to_placement_aggregate=true we nolonger need the avaiablity zone filter right | |
| 13:06:35 | sean-k-mooney | have we talked about when we might deprecate that and swap over to the placment way of doing it by default | |
| 13:06:46 | gibi | sean-k-mooney: only aggregate info is in placeent but not AZ info | |
| 13:07:35 | kashyap | sean-k-mooney: libvirt doesn't do any automatic memory ballooning, does it? | |
| 13:07:45 | gibi | I mean the aggregate metadata is not in placement and the AZ name is in the aggregate metadata | |
| 13:07:48 | sean-k-mooney | right but what i ment is could we start enabling the prefilter by default and deprecate the filter for removal | |
| 13:08:06 | sean-k-mooney | gibi: specifclaly this prefilter https://github.com/openstack/nova/blob/8988316b8c132c9662dea6cf0345975e87ce7344/nova/scheduler/request_filter.py#L136-L163 | |
| 13:08:18 | sean-k-mooney | kashyap: no it does not | |
| 13:08:50 | kashyap | sean-k-mooney: Then what do you mean by: "i know that at least one user wants us to remove the automatic memory balloan that is added by livirt" | |
| 13:08:52 | sean-k-mooney | kashyap: you can makrk the domain to try an deflate on OOM but even then im not sure that it will work without something in the guest | |
| 13:09:24 | sean-k-mooney | kashyap: libvirt adds a virtio-ballon device even if you dont ask for one by default | |
| 13:09:30 | kashyap | That "deflate-on-OOM" (which is a QEMU feature) is only for the guest to reclaim its mem back - nothing related to host | |
| 13:09:37 | sean-k-mooney | kashyap: we had one user ask for a way to disable that | |
| 13:09:43 | kashyap | sean-k-mooney: Ah, merely the adding of the ballooning device | |
| 13:09:44 | gibi | sean-k-mooney: I see now. Yes it make sense. I think we can deprecate the Filter and user the perfilter + placement | |
| 13:09:55 | kashyap | sean-k-mooney: Yes, I know - the real-time usecase doesn't want it | |
| 13:10:06 | sean-k-mooney | kashyap: actrully it was not for realtime | |
| 13:10:36 | sean-k-mooney | kashyap: it was for a guest with 2TB of ram and the debian folk were wondering if that was causeing some qemu crashes | |
| 13:10:54 | sean-k-mooney | but i actully think it was related to the adress space not the memory ballaon | |
| 13:11:20 | sean-k-mooney | gibi: ok ill file a blueprint and submit a patch | |
| 13:11:41 | stephenfin | gibi: replied on https://review.opendev.org/#/c/733668/1 | |
| 13:11:54 | sean-k-mooney | gibi: i was just looking at my local.conf an noticed i was still enebling the retry filter which has notbeen needed for a few release and notice i was also enabling the az filter | |
| 13:13:45 | openstackgerrit | Merged openstack/nova master: [Nova] Add reference to Placement installation guide https://review.opendev.org/726936 | |
| 13:14:28 | sean-k-mooney | kashyap: the issue they were hiting is by default qemu was not allcating enough virtual adressspace to create a guest larger than 2TB | |
| 13:20:26 | kashyap | sean-k-mooney: I see | |
| 13:20:31 | kashyap | Thx for the context | |
| 13:21:35 | openstack | bugzilla.redhat.com bug 630975 in seabios "KVM guest limited to 40bit of physical address space" [Medium,Closed: errata] - Assigned to aarcange | |
| 13:21:35 | sean-k-mooney | kashyap: ya they just wanted to turn it off for debuging reasons they were hitting basically this bug https://bugzilla.redhat.com/show_bug.cgi?id=630975 or https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1769053 | |
| 13:21:36 | openstack | Launchpad bug 1769053 in libvirt (Ubuntu) "Ability to control phys-bits through libvirt" [High,Triaged] | |
| 13:21:57 | sean-k-mooney | i wonder if nova shoudl actully be setting the libvirt option | |
| 13:23:43 | sean-k-mooney | kashyap: they had to create a custom machine type to work around the issue | |
| 13:24:40 | openstack | bugzilla.redhat.com bug 1578278 in libvirt "Ability to control phys-bits through libvirt" [Unspecified,New] - Assigned to libvirt-maint | |
| 13:24:40 | sean-k-mooney | hum i guess this is still open https://bugzilla.redhat.com/show_bug.cgi?id=1578278 | |
| 13:24:52 | sean-k-mooney | so maybe this can only be done at the qemu level | |
| 13:28:21 | gibi | sean-k-mooney: cool thanks. | |
| 13:28:29 | gibi | stephenfin: looking... | |
| 13:34:22 | gibi | stephenfin: sorry. totally my bad I guess I wasn't paying enough attention. Sorry again. | |
| 13:34:38 | stephenfin | it's all good :) | |
| 13:35:12 | gibi | :) | |
| 13:39:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use 'Exception.__traceback__' for versioned notifications https://review.opendev.org/733668 | |
| 13:39:10 | stephenfin | gibi: actually, looks like I had a pep8 failure on that ^ Trying to figure out why pre-commit didn't catch it | |
| 13:40:35 | sean-k-mooney | stephenfin: was it a hacking rule im not sure pre-commit runs those | |
| 13:40:49 | sean-k-mooney | it might i just have not checked | |
| 13:41:03 | stephenfin | not a hacking rule, no, but pre-commit does run hacking anyway | |
| 13:47:49 | gibi | stephenfin: ack, reapplied my +2 | |
| 13:47:55 | stephenfin | ta | |
| 13:50:46 | openstackgerrit | Merged openstack/os-resource-classes master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/731512 | |
| 14:43:26 | sean-k-mooney | frickler: so im not seeing any detach errors but i am seeing this http://paste.openstack.org/show/794474/ | |
| 14:45:38 | frickler | sean-k-mooney: yeah, that's related to eventlet and py>=3.7 it seems, see my note in the bug and the other bug linked. I'm not sure whether that's related or not, haven't been able to build an easier reproducer so far, either | |
| 14:45:58 | bauzas | frickler: we don't support yet py>=3.7 | |
| 14:46:36 | sean-k-mooney | bauzas: well we kindo of do. there is an non voting job we can enable and i think its planned for victoria | |
| 14:47:08 | sean-k-mooney | yep it is https://opendev.org/openstack/governance/src/branch/master/reference/runtimes/victoria.rst#user-content-python-runtimes-for-victoria | |
| 14:47:45 | sean-k-mooney | so 3.6 and 3.8 will be the required supported version once we swap form 18.04 | |
| 14:48:00 | sean-k-mooney | to 20.04 | |
| 14:48:57 | sean-k-mooney | frickler: the eventlet thing seams to be intermitent | |
| 14:49:08 | sean-k-mooney | i see the command are sometimes passing fine | |
| 14:52:05 | sean-k-mooney | frickler: given i deployed master i would apper that https://review.opendev.org/#/c/707474/ does not actully work | |
| 14:53:33 | frickler | sean-k-mooney: you might want to test with a revert, but I was assuming that it would be only incomplete and we would have much more issues without it | |
| 14:54:05 | sean-k-mooney | yes it likely incomplete but i can try that quickly. | |
| 14:54:18 | sean-k-mooney | me might need to do this in oslo as well | |
| 15:00:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM: Test the state of VMware NSX 3pp CI https://review.opendev.org/734114 | |
| 19:19:45 | dansmith | sean-k-mooney: around? | |
| 19:28:08 | sean-k-mooney | dansmith: yes | |
| 19:28:18 | sean-k-mooney | just traging a networking issue how can i help | |
| 19:28:44 | dansmith | sean-k-mooney: I want to create a variant of the ceph job that has multiple glance stores, so some glance conf tweaking | |
| 19:28:56 | dansmith | sean-k-mooney: I think I need a devstack_local_conf section in the job def in .zuul.conf yeah? | |
| 19:29:18 | dansmith | I'm not quite sure what it needs to look like, can I pastebin something in a sec for you to look at? | |
| 19:29:38 | sean-k-mooney | so you likely need to set different values on the contoler/all-in-one node vs the compute only node | |
| 19:29:49 | sean-k-mooney | so you will need to set different value in different groups | |
| 19:30:07 | sean-k-mooney | dansmith: sure give me the linke and ill take a look | |
| 19:30:41 | dansmith | this can be single-node fwiw | |
| 19:30:58 | sean-k-mooney | with multiple stores? | |
| 19:31:01 | dansmith | yup | |
| 19:31:20 | sean-k-mooney | oh are you configuring mutile glance backends or something | |