| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 20:53:33 | openstackgerrit | Eric Fried proposed openstack/nova master: Invalidate cache when _refresh_associations fails https://review.opendev.org/678960 | |
| 20:53:34 | efried | mriedem: This ^ may be sufficient for bug 1841481, assuming "the next refresh" is quick enough for us to repopulate the cache. Otherwise we may wish to wrap _refresh_associations in a one-loop @retry (but not on ClientException). Let me know your thoughts if you please. | |
| 20:53:35 | openstack | bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,Triaged] https://launchpad.net/bugs/1841481 - Assigned to Eric Fried (efried) | |
| 20:55:07 | zigo | sean-k-mooney: Got a 6 core Xeon (so 12 threads...)... | |
| 20:56:08 | sean-k-mooney | avoids the whos system is faster trap. | |
| 20:59:26 | zigo | The nice thing is that I do have enough NVME space to have a full Debian mirror ... | |
| 20:59:36 | zigo | deb file:///home/ftp/debian ... :P | |
| 21:00:08 | sean-k-mooney | thats like somewhere betten 230-400GB right | |
| 21:00:33 | sean-k-mooney | i used to use apt-cacheng to cache the things i actully was using | |
| 21:00:39 | sean-k-mooney | i should set that up again. | |
| 21:00:58 | zigo | sean-k-mooney: 463 GB right now, only amd64 + sources, all available suites. | |
| 21:01:29 | zigo | The point is, it directly fetches on drive, avoiding network socket indirections... | |
| 21:01:39 | sean-k-mooney | right we had a ubunut mirror for the intel nfv ci and i rememebr it was somere in that range | |
| 21:01:44 | zigo | sean-k-mooney: I used approx. | |
| 21:01:47 | sean-k-mooney | but that was like 3 years ago | |
| 21:01:54 | sean-k-mooney | so i assuemd it grew | |
| 21:03:17 | mriedem | melwitt: time for you to bug dansmith https://review.opendev.org/#/c/507486/69 | |
| 21:03:41 | sean-k-mooney | prometheanfire: do you know if the lxml thing only happins on a specific python verions | |
| 21:03:58 | sean-k-mooney | im running the test uder py36 at the momemt | |
| 21:04:03 | mriedem | zigo: are you testing on py27 or py36? | |
| 21:04:34 | sean-k-mooney | i can test both but py36 is faster for me to test with | |
| 21:04:57 | sean-k-mooney | i have 4 test failrue on py36 | |
| 21:06:09 | sean-k-mooney | all of which are not using our xml matcher | |
| 21:06:27 | sean-k-mooney | there doing self.assertEqual(diska_xml.strip(), actual_diska_xml.strip()) | |
| 21:06:42 | sean-k-mooney | or assert called with | |
| 21:08:21 | prometheanfire | sean-k-mooney: I don't know :( | |
| 21:08:35 | sean-k-mooney | it broke on py36 so i can reporduce | |
| 21:08:40 | sean-k-mooney | so its fine | |
| 21:08:41 | zigo | mriedem: python 3.7 | |
| 21:09:12 | zigo | sean-k-mooney: py37 | |
| 21:09:23 | sean-k-mooney | it likely not python specfic | |
| 21:09:28 | zigo | Right. | |
| 21:11:08 | sean-k-mooney | actully at least one of the tests fail on py3 and pass on py27 | |
| 21:11:18 | sean-k-mooney | so maybe it is but ill fix it for both in anycase | |
| 21:14:25 | donnyd | so here is some interesting information about the image cache | |
| 21:14:37 | donnyd | I keep getting this error thrown | |
| 21:14:53 | donnyd | https://www.irccloud.com/pastebin/9eB2NYg4/ | |
| 21:15:08 | sean-k-mooney | ya i have seen that | |
| 21:15:11 | donnyd | most notably this nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d0 | |
| 21:15:32 | donnyd | however that image is there, and is continually downloaded over and over | |
| 21:15:50 | sean-k-mooney | i think mdbooth was fixing a cese were if we fail to resize an image we delete the base image backing file in the cache | |
| 21:16:20 | donnyd | https://www.irccloud.com/pastebin/yllMTx02/ | |
| 21:17:17 | donnyd | its not a resize | |
| 21:17:21 | donnyd | -rw-r--r-- 1 nova nova 0 Aug 27 21:17 1675f8d33da6cfdd6354d2078b61c3c11ff417d0.part | |
| 21:17:21 | donnyd | -rw-r--r-- 1 libvirt-qemu kvm 7054688256 Aug 27 21:13 1675f8d33da6cfdd6354d2078b61c3c11ff417d0 | |
| 21:17:50 | mriedem | wonder if there is an access issue? | |
| 21:18:03 | mriedem | the DiskNotFound might just be a shitty generic exception that is thrown | |
| 21:18:05 | mriedem | *raised | |
| 21:18:59 | donnyd | well its owned by nova till download and then changed to libvirtd-qemu | |
| 21:19:27 | mriedem | raising from here though so not an access issue https://github.com/openstack/nova/blob/master/nova/virt/images.py#L58 | |
| 21:20:49 | donnyd | If I mount the whole directory for nova/instances this error goes away | |
| 21:21:47 | sean-k-mooney | donnyd: normally nova is in the libvirt group | |
| 21:22:42 | sean-k-mooney | so nova should be able to read and write to image owned by nova | |
| 21:22:45 | donnyd | Well there isnt an issue if I purge the cache, it seems to download and then run with it fine | |
| 21:22:49 | sean-k-mooney | *libvirt/qemu | |
| 21:23:23 | donnyd | But each hypervisor that tries to read this image throws that error and then redownloads | |
| 21:27:35 | sean-k-mooney | the way the xml match works is dumb/ not documented | |
| 21:29:52 | sean-k-mooney | why do retrun nothing mean it matched | |
| 21:30:27 | sean-k-mooney | now that i know that i can actully use it | |
| 21:30:59 | sean-k-mooney | oh this comes from testtools... | |
| 21:32:14 | donnyd | well its reporting back exactly what qemu-img is telling it | |
| 21:32:57 | sean-k-mooney | donnyd: what os are you on | |
| 21:33:06 | donnyd | qemu-img: Could not open '/var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d0' | |
| 21:33:06 | donnyd | sudo qemu-img info /var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d0 | |
| 21:33:11 | donnyd | ubuntu 18.04 | |
| 21:33:13 | sean-k-mooney | can you check if apparmor or systmed is blockign it | |
| 21:33:49 | sean-k-mooney | e.g. check dmesg | |
| 21:34:52 | donnyd | possible... | |
| 21:35:09 | donnyd | not sure what i am looking for apparmor to say | |
| 21:36:06 | sean-k-mooney | something like this | |
| 21:36:08 | sean-k-mooney | audit: type=1400 audit(1565883657.055:70): apparmor="ALLOWED" operation="open" profile="libreoffice-soffice" name="/proc/31189/comm" pid=31189 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 | |
| 21:36:33 | sean-k-mooney | exceet you are looking for apparmor="DENIED" | |
| 21:37:22 | donnyd | when i grep fro DENIED i return nothing | |
| 21:37:39 | sean-k-mooney | its proably not an apparmor thing but it could be if the file permession allow you to read the file as root | |
| 21:38:32 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 21:38:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | |
| 21:38:33 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 21:38:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 21:38:34 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 21:38:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor https://review.opendev.org/678887 | |
| 21:38:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 21:46:50 | mriedem | sean-k-mooney: https://blueprints.launchpad.net/nova/+spec/image-metadata-prefiltering is now in a runway | |
| 21:47:12 | sean-k-mooney | mriedem: ok | |
| 21:49:53 | sean-k-mooney | the nova-next failure in the middle patch looks unrelated so i just rechecked it | |
| 21:49:58 | sean-k-mooney | the rest are clean | |
| 21:50:11 | sean-k-mooney | if you see anything ill adress it tomorow | |
| 22:13:13 | sean-k-mooney | prometheanfire: i think i have fixed the lxml issue locally | |
| 22:15:29 | sean-k-mooney | fungi: ya that looks fine | |
| 22:15:46 | sean-k-mooney | * cve description | |
| 22:16:13 | fungi | thanks again sean-k-mooney! | |
| 22:25:16 | openstackgerrit | sean mooney proposed openstack/nova master: fix lxml compatiblity issues https://review.opendev.org/678964 | |
| 22:25:42 | sean-k-mooney | zigo: efried prometheanfire ^ | |
| 22:25:59 | sean-k-mooney | i think that should fix it | |
| 22:27:04 | zigo | sean-k-mooney: Nice ! :) | |
| 22:27:23 | zigo | I've already blacklisted the unit tests in my last upload to Sid though, but if it passes, I'll add your patch instead. | |
| 22:29:47 | efried | sean-k-mooney: assertXmlEqual is the one-step alias for what you're doing in there. | |
| 22:30:08 | sean-k-mooney | yep | |
| 22:30:30 | sean-k-mooney | i knew that was a thing too but blanked when writing it | |
| 22:30:37 | sean-k-mooney | but ill fix that tommrow | |
| 22:37:20 | sean-k-mooney | have people seen "ValueError: invalid literal for int() with base 10: '12.1'" | |