| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 20:47:20 | sean-k-mooney | ya i delete that after the hard reboot it was running on cloud-3 with the device claimed on cloud-4 so that is not good. | |
| 20:47:36 | sean-k-mooney | im going to unstack and call it a night. | |
| 20:47:37 | efried | mriedem: yes, sean-k-mooney was looking into it. | |
| 20:47:46 | sean-k-mooney | mriedem: ya im ment to be looking into that | |
| 20:47:49 | efried | had to do with attribute ordering. | |
| 20:47:56 | mriedem | yup https://bugs.launchpad.net/nova/+bug/1841667 | |
| 20:47:56 | sean-k-mooney | it was what i had planned to do today... | |
| 20:47:58 | openstack | Launchpad bug 1841667 in OpenStack Compute (nova) stein "failing libvirt tests: need ordering" [Undecided,New] | |
| 20:48:02 | zigo | mriedem: 4.4.1. | |
| 20:48:18 | mriedem | efried: sean-k-mooney: did we have a bug for it? | |
| 20:48:19 | zigo | That's kind of new in Sid... | |
| 20:48:29 | sean-k-mooney | there was one filed yes | |
| 20:48:40 | sean-k-mooney | mriedem: prometheanfire filed it i think | |
| 20:48:48 | openstack | Launchpad bug 1838666 in OpenStack Compute (nova) "lxml 4.4.0 causes failed tests in nova" [Undecided,Confirmed] | |
| 20:48:48 | mriedem | https://bugs.launchpad.net/nova/+bug/1838666 | |
| 20:48:50 | mriedem | got it | |
| 20:49:00 | zigo | mriedem: Uploaded 10 days ago: https://tracker.debian.org/pkg/lxml | |
| 20:51:33 | zigo | Wow, with my brand new laptop, it only takes me 5 minutes to run all Nova unit tests now! :) | |
| 20:51:51 | mriedem | sounds like a challenge | |
| 20:53:10 | sean-k-mooney | ya its nice when you can throw more cores at it and it scales well | |
| 20:53:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Raise when inventory retrieval & refresh fails https://review.opendev.org/678957 | |
| 20:53:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove @safe_connect from _get_sharing_providers https://review.opendev.org/678959 | |
| 20:53:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove @safe_connect from _get_provider_aggregates https://review.opendev.org/678958 | |
| 20:53:33 | openstackgerrit | Eric Fried proposed openstack/nova master: Invalidate cache when _refresh_associations fails https://review.opendev.org/678960 | |
| 20:53:34 | efried | mriedem: This ^ may be sufficient for bug 1841481, assuming "the next refresh" is quick enough for us to repopulate the cache. Otherwise we may wish to wrap _refresh_associations in a one-loop @retry (but not on ClientException). Let me know your thoughts if you please. | |
| 20:53:35 | openstack | bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,Triaged] https://launchpad.net/bugs/1841481 - Assigned to Eric Fried (efried) | |
| 20:55:07 | zigo | sean-k-mooney: Got a 6 core Xeon (so 12 threads...)... | |
| 20:56:08 | sean-k-mooney | avoids the whos system is faster trap. | |
| 20:59:26 | zigo | The nice thing is that I do have enough NVME space to have a full Debian mirror ... | |
| 20:59:36 | zigo | deb file:///home/ftp/debian ... :P | |
| 21:00:08 | sean-k-mooney | thats like somewhere betten 230-400GB right | |
| 21:00:33 | sean-k-mooney | i used to use apt-cacheng to cache the things i actully was using | |
| 21:00:39 | sean-k-mooney | i should set that up again. | |
| 21:00:58 | zigo | sean-k-mooney: 463 GB right now, only amd64 + sources, all available suites. | |
| 21:01:29 | zigo | The point is, it directly fetches on drive, avoiding network socket indirections... | |
| 21:01:39 | sean-k-mooney | right we had a ubunut mirror for the intel nfv ci and i rememebr it was somere in that range | |
| 21:01:44 | zigo | sean-k-mooney: I used approx. | |
| 21:01:47 | sean-k-mooney | but that was like 3 years ago | |
| 21:01:54 | sean-k-mooney | so i assuemd it grew | |
| 21:03:17 | mriedem | melwitt: time for you to bug dansmith https://review.opendev.org/#/c/507486/69 | |
| 21:03:41 | sean-k-mooney | prometheanfire: do you know if the lxml thing only happins on a specific python verions | |
| 21:03:58 | sean-k-mooney | im running the test uder py36 at the momemt | |
| 21:04:03 | mriedem | zigo: are you testing on py27 or py36? | |
| 21:04:34 | sean-k-mooney | i can test both but py36 is faster for me to test with | |
| 21:04:57 | sean-k-mooney | i have 4 test failrue on py36 | |
| 21:06:09 | sean-k-mooney | all of which are not using our xml matcher | |
| 21:06:27 | sean-k-mooney | there doing self.assertEqual(diska_xml.strip(), actual_diska_xml.strip()) | |
| 21:06:42 | sean-k-mooney | or assert called with | |
| 21:08:21 | prometheanfire | sean-k-mooney: I don't know :( | |
| 21:08:35 | sean-k-mooney | it broke on py36 so i can reporduce | |
| 21:08:40 | sean-k-mooney | so its fine | |
| 21:08:41 | zigo | mriedem: python 3.7 | |
| 21:09:12 | zigo | sean-k-mooney: py37 | |
| 21:09:23 | sean-k-mooney | it likely not python specfic | |
| 21:09:28 | zigo | Right. | |
| 21:11:08 | sean-k-mooney | actully at least one of the tests fail on py3 and pass on py27 | |
| 21:11:18 | sean-k-mooney | so maybe it is but ill fix it for both in anycase | |
| 21:14:25 | donnyd | so here is some interesting information about the image cache | |
| 21:14:37 | donnyd | I keep getting this error thrown | |
| 21:14:53 | donnyd | https://www.irccloud.com/pastebin/9eB2NYg4/ | |
| 21:15:08 | sean-k-mooney | ya i have seen that | |
| 21:15:11 | donnyd | most notably this nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d0 | |
| 21:15:32 | donnyd | however that image is there, and is continually downloaded over and over | |
| 21:15:50 | sean-k-mooney | i think mdbooth was fixing a cese were if we fail to resize an image we delete the base image backing file in the cache | |
| 21:16:20 | donnyd | https://www.irccloud.com/pastebin/yllMTx02/ | |
| 21:17:17 | donnyd | its not a resize | |
| 21:17:21 | donnyd | -rw-r--r-- 1 nova nova 0 Aug 27 21:17 1675f8d33da6cfdd6354d2078b61c3c11ff417d0.part | |
| 21:17:21 | donnyd | -rw-r--r-- 1 libvirt-qemu kvm 7054688256 Aug 27 21:13 1675f8d33da6cfdd6354d2078b61c3c11ff417d0 | |
| 21:17:50 | mriedem | wonder if there is an access issue? | |
| 21:18:03 | mriedem | the DiskNotFound might just be a shitty generic exception that is thrown | |
| 21:18:05 | mriedem | *raised | |
| 21:18:59 | donnyd | well its owned by nova till download and then changed to libvirtd-qemu | |
| 21:19:27 | mriedem | raising from here though so not an access issue https://github.com/openstack/nova/blob/master/nova/virt/images.py#L58 | |
| 21:20:49 | donnyd | If I mount the whole directory for nova/instances this error goes away | |
| 21:21:47 | sean-k-mooney | donnyd: normally nova is in the libvirt group | |
| 21:22:42 | sean-k-mooney | so nova should be able to read and write to image owned by nova | |
| 21:22:45 | donnyd | Well there isnt an issue if I purge the cache, it seems to download and then run with it fine | |
| 21:22:49 | sean-k-mooney | *libvirt/qemu | |
| 21:23:23 | donnyd | But each hypervisor that tries to read this image throws that error and then redownloads | |
| 21:27:35 | sean-k-mooney | the way the xml match works is dumb/ not documented | |
| 21:29:52 | sean-k-mooney | why do retrun nothing mean it matched | |
| 21:30:27 | sean-k-mooney | now that i know that i can actully use it | |
| 21:30:59 | sean-k-mooney | oh this comes from testtools... | |
| 21:32:14 | donnyd | well its reporting back exactly what qemu-img is telling it | |
| 21:32:57 | sean-k-mooney | donnyd: what os are you on | |
| 21:33:06 | donnyd | qemu-img: Could not open '/var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d0' | |
| 21:33:06 | donnyd | sudo qemu-img info /var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d0 | |
| 21:33:11 | donnyd | ubuntu 18.04 | |
| 21:33:13 | sean-k-mooney | can you check if apparmor or systmed is blockign it | |
| 21:33:49 | sean-k-mooney | e.g. check dmesg | |
| 21:34:52 | donnyd | possible... | |
| 21:35:09 | donnyd | not sure what i am looking for apparmor to say | |
| 21:36:06 | sean-k-mooney | something like this | |
| 21:36:08 | sean-k-mooney | audit: type=1400 audit(1565883657.055:70): apparmor="ALLOWED" operation="open" profile="libreoffice-soffice" name="/proc/31189/comm" pid=31189 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 | |
| 21:36:33 | sean-k-mooney | exceet you are looking for apparmor="DENIED" | |
| 21:37:22 | donnyd | when i grep fro DENIED i return nothing | |
| 21:37:39 | sean-k-mooney | its proably not an apparmor thing but it could be if the file permession allow you to read the file as root | |
| 21:38:32 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 21:38:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | |
| 21:38:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |