| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-08 | |||
| 10:40:18 | stephenfin | I suspect so too | |
| 10:40:26 | stephenfin | Probably someone deploying things and finding the gaps | |
| 10:40:50 | sean-k-mooney | looks like ther other patches were to the sandbox project so ya likely. | |
| 10:41:06 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Encrypted Emulated Virtual TPM https://review.opendev.org/728505 | |
| 11:12:36 | stephenfin | bauzas: Can I get you to look at https://review.opendev.org/728505 this week, please? | |
| 11:12:55 | stephenfin | bauzas: I also have a couple of code reviews I'd like some eyes on. Happy to horse trade ;) | |
| 11:52:40 | bauzas | sean-k-mooney: stephenfin: sorry was at lunch | |
| 11:59:28 | bauzas | stephenfin: sure, I wanna do some specs reviews this week anyway | |
| 12:21:15 | openstack | Launchpad bug 1882521 in OpenStack Compute (nova) "Failing device detachments on Focal" [Undecided,New] | |
| 12:21:15 | frickler | so if anyone is interesting in getting devstack testing on focal to proceed, I created this bug report. lyarwood had started to look at it earlier, I don't have enough clue myself, in the end it might be nova, libvirt, qemu, cinder or even tempest is the actual culprit https://bugs.launchpad.net/nova/+bug/1882521 | |
| 12:23:04 | sean-k-mooney | frickler: are there any known blockers right now | |
| 12:23:40 | sean-k-mooney | frickler: ill be setting up a devenv for numa stuff later today and was considering if i should try centos 8 or ubuntu 20.04 instaed of sticking with 18.04 | |
| 12:24:36 | sean-k-mooney | Device detach failed for vdb: Unable to detach the device from the live config. | |
| 12:24:53 | sean-k-mooney | frickler: that looks like its comming from libvirt althoug hi have not looked at the code yet | |
| 12:25:23 | sean-k-mooney | frickler: was the default lvm backend or ceph? | |
| 12:28:02 | frickler | sean-k-mooney: in devstack it is lvm, although I tested locally with ceph and see similar issues. but there's also a different bug for that, one moment | |
| 12:28:03 | kashyap | sean-k-mooney: Hey, do you know of any prior rejected work on memory ballooning? | |
| 12:28:06 | kashyap | ... in Nova | |
| 12:28:37 | frickler | sean-k-mooney: also, I think that those test-failures are blocking for focal, I wouldn't want to merge focal support with them blacklisted | |
| 12:28:49 | sean-k-mooney | not specifcally but i know dansmit is not a fan of nova managing it | |
| 12:29:16 | sean-k-mooney | frickler: oh i ment in terms of me swaping my daily dev env to focal rather then the ci | |
| 12:29:36 | sean-k-mooney | frickler: if i start using it and hit error its more likely that im going to fix them | |
| 12:30:55 | sean-k-mooney | kashyap: i know that at least one user wants us to remove the automatic memory balloan that is added by livirt | |
| 12:31:22 | sean-k-mooney | kashyap: i also know that we are getting asked to consider added memory balloan managemnt downstream | |
| 12:32:58 | frickler | sean-k-mooney: ah, I see, you'd need to use my patch https://review.opendev.org/704831 , apart from the tempest failures it has been running pretty well for me for some months now | |
| 12:34:01 | sean-k-mooney | frickler: cool i was planning on swapping to it for the numa work im starting. the vms i have created are already on it but i had not got as far as stacking yey | |
| 12:34:49 | sean-k-mooney | so if i use that devstack patch it should be good | |
| 12:34:51 | frickler | sean-k-mooney: the ceph thing seems to have been fixed with https://review.opendev.org/725812 , not sure I've tested with that patch in tempest | |
| 12:35:13 | frickler | sean-k-mooney: yes, that would be great, let me know if you find any further issues | |
| 12:36:01 | sean-k-mooney | ill set up the multi node env and before i start modifying code ill try and do a few volume tests i assuem you are hitting the issue locally too? | |
| 12:36:08 | sean-k-mooney | e.g. its not just in the ci | |
| 12:36:43 | frickler | sean-k-mooney: yes, although locally I didn't see it 100%, might be a bit load/memory pressure related | |
| 12:37:21 | sean-k-mooney | hehe its funny but i realised the other day i finally got https://opendev.org/opentack into my muscel memory instead of defaulting to github | |
| 12:38:18 | sean-k-mooney | frickler: so while your here | |
| 12:38:21 | sean-k-mooney | ubuntu@numa-1:/opt/repos/devstack$ python3 -m pip install --user git-review | |
| 12:38:23 | sean-k-mooney | /usr/bin/python3: No module named pip | |
| 12:38:35 | sean-k-mooney | am i imagining things or is pip not part of the standard libary now | |
| 12:38:41 | sean-k-mooney | i know how to fix this | |
| 12:38:55 | sean-k-mooney | but im just wondering why on 3.8 i have to install a seperate package | |
| 12:39:05 | sean-k-mooney | i tought it was ment to be always present | |
| 12:39:47 | frickler | yeah, they did strange things there, same for venv, no idea why. for git-review, I just use the pkg from universe, not pip | |
| 12:40:26 | sean-k-mooney | i have had issue on centos7 with it being old and busted | |
| 12:40:37 | sean-k-mooney | so i default to pip for it | |
| 12:41:04 | sean-k-mooney | mainly to have a consitent behavior across distos to be honest | |
| 12:43:30 | frickler | I also had to create a symlink for pip3.8 since the distro pkg only provides pip3, but devstack wants it version specific | |
| 12:43:43 | sean-k-mooney | frickler: i think they are treating pip and venv as dev tools | |
| 12:44:27 | sean-k-mooney | frickler: we proably should modify devstack to not use the pip command but instaead do it the python3 -m pip way | |
| 12:45:05 | sean-k-mooney | it will avoid that issue but also i think that is the new recomendation to ensure it installs where you think it will | |
| 12:45:34 | sean-k-mooney | obviously python3 can be python3.8 or the version specifc one too | |
| 13:06:13 | sean-k-mooney | gibi: bauzas if we set map_az_to_placement_aggregate=true we nolonger need the avaiablity zone filter right | |
| 13:06:35 | sean-k-mooney | have we talked about when we might deprecate that and swap over to the placment way of doing it by default | |
| 13:06:46 | gibi | sean-k-mooney: only aggregate info is in placeent but not AZ info | |
| 13:07:35 | kashyap | sean-k-mooney: libvirt doesn't do any automatic memory ballooning, does it? | |
| 13:07:45 | gibi | I mean the aggregate metadata is not in placement and the AZ name is in the aggregate metadata | |
| 13:07:48 | sean-k-mooney | right but what i ment is could we start enabling the prefilter by default and deprecate the filter for removal | |
| 13:08:06 | sean-k-mooney | gibi: specifclaly this prefilter https://github.com/openstack/nova/blob/8988316b8c132c9662dea6cf0345975e87ce7344/nova/scheduler/request_filter.py#L136-L163 | |
| 13:08:18 | sean-k-mooney | kashyap: no it does not | |
| 13:08:50 | kashyap | sean-k-mooney: Then what do you mean by: "i know that at least one user wants us to remove the automatic memory balloan that is added by livirt" | |
| 13:08:52 | sean-k-mooney | kashyap: you can makrk the domain to try an deflate on OOM but even then im not sure that it will work without something in the guest | |
| 13:09:24 | sean-k-mooney | kashyap: libvirt adds a virtio-ballon device even if you dont ask for one by default | |
| 13:09:30 | kashyap | That "deflate-on-OOM" (which is a QEMU feature) is only for the guest to reclaim its mem back - nothing related to host | |
| 13:09:37 | sean-k-mooney | kashyap: we had one user ask for a way to disable that | |
| 13:09:43 | kashyap | sean-k-mooney: Ah, merely the adding of the ballooning device | |
| 13:09:44 | gibi | sean-k-mooney: I see now. Yes it make sense. I think we can deprecate the Filter and user the perfilter + placement | |
| 13:09:55 | kashyap | sean-k-mooney: Yes, I know - the real-time usecase doesn't want it | |
| 13:10:06 | sean-k-mooney | kashyap: actrully it was not for realtime | |
| 13:10:36 | sean-k-mooney | kashyap: it was for a guest with 2TB of ram and the debian folk were wondering if that was causeing some qemu crashes | |
| 13:10:54 | sean-k-mooney | but i actully think it was related to the adress space not the memory ballaon | |
| 13:11:20 | sean-k-mooney | gibi: ok ill file a blueprint and submit a patch | |
| 13:11:41 | stephenfin | gibi: replied on https://review.opendev.org/#/c/733668/1 | |
| 13:11:54 | sean-k-mooney | gibi: i was just looking at my local.conf an noticed i was still enebling the retry filter which has notbeen needed for a few release and notice i was also enabling the az filter | |
| 13:13:45 | openstackgerrit | Merged openstack/nova master: [Nova] Add reference to Placement installation guide https://review.opendev.org/726936 | |
| 13:14:28 | sean-k-mooney | kashyap: the issue they were hiting is by default qemu was not allcating enough virtual adressspace to create a guest larger than 2TB | |
| 13:20:26 | kashyap | sean-k-mooney: I see | |
| 13:20:31 | kashyap | Thx for the context | |
| 13:21:35 | openstack | bugzilla.redhat.com bug 630975 in seabios "KVM guest limited to 40bit of physical address space" [Medium,Closed: errata] - Assigned to aarcange | |
| 13:21:35 | sean-k-mooney | kashyap: ya they just wanted to turn it off for debuging reasons they were hitting basically this bug https://bugzilla.redhat.com/show_bug.cgi?id=630975 or https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1769053 | |
| 13:21:36 | openstack | Launchpad bug 1769053 in libvirt (Ubuntu) "Ability to control phys-bits through libvirt" [High,Triaged] | |
| 13:21:57 | sean-k-mooney | i wonder if nova shoudl actully be setting the libvirt option | |
| 13:23:43 | sean-k-mooney | kashyap: they had to create a custom machine type to work around the issue | |
| 13:24:40 | openstack | bugzilla.redhat.com bug 1578278 in libvirt "Ability to control phys-bits through libvirt" [Unspecified,New] - Assigned to libvirt-maint | |
| 13:24:40 | sean-k-mooney | hum i guess this is still open https://bugzilla.redhat.com/show_bug.cgi?id=1578278 | |
| 13:24:52 | sean-k-mooney | so maybe this can only be done at the qemu level | |
| 13:28:21 | gibi | sean-k-mooney: cool thanks. | |
| 13:28:29 | gibi | stephenfin: looking... | |
| 13:34:22 | gibi | stephenfin: sorry. totally my bad I guess I wasn't paying enough attention. Sorry again. | |
| 13:34:38 | stephenfin | it's all good :) | |
| 13:35:12 | gibi | :) | |
| 13:39:01 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use 'Exception.__traceback__' for versioned notifications https://review.opendev.org/733668 | |
| 13:39:10 | stephenfin | gibi: actually, looks like I had a pep8 failure on that ^ Trying to figure out why pre-commit didn't catch it | |
| 13:40:35 | sean-k-mooney | stephenfin: was it a hacking rule im not sure pre-commit runs those | |
| 13:40:49 | sean-k-mooney | it might i just have not checked | |
| 13:41:03 | stephenfin | not a hacking rule, no, but pre-commit does run hacking anyway | |
| 13:47:49 | gibi | stephenfin: ack, reapplied my +2 | |
| 13:47:55 | stephenfin | ta | |
| 13:50:46 | openstackgerrit | Merged openstack/os-resource-classes master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/731512 | |
| 14:43:26 | sean-k-mooney | frickler: so im not seeing any detach errors but i am seeing this http://paste.openstack.org/show/794474/ | |
| 14:45:38 | frickler | sean-k-mooney: yeah, that's related to eventlet and py>=3.7 it seems, see my note in the bug and the other bug linked. I'm not sure whether that's related or not, haven't been able to build an easier reproducer so far, either | |
| 14:45:58 | bauzas | frickler: we don't support yet py>=3.7 | |
| 14:46:36 | sean-k-mooney | bauzas: well we kindo of do. there is an non voting job we can enable and i think its planned for victoria | |