| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 20:18:29 | sean-k-mooney | yes that is true | |
| 20:19:52 | mriedem | i'm not even sure what you'd test after the clobbered live migration to show it fails, or how we'd recreate that in a functional test to prevent future regressions, but i know that having sean manually test all these weird things isn't sustainable | |
| 20:20:22 | dansmith | obviously | |
| 20:20:25 | mriedem | i guess you could like cold migrate back to the source host and verify there are any claimed pci devices for the instance | |
| 20:21:03 | sean-k-mooney | the pci device are still tracked in the pci tacker as used by the instace | |
| 20:21:11 | mriedem | oh great | |
| 20:21:18 | sean-k-mooney | actully i should check that | |
| 20:21:19 | mriedem | didn't we just talk about something like this...vpmem? | |
| 20:21:49 | sean-k-mooney | kind of | |
| 20:23:41 | sean-k-mooney | yes just check we are correctly tracking thing at the host level in the pci tracker at least | |
| 20:25:35 | donnyd | mriedem: melwitt I have the image cache dir all setup and running | |
| 20:25:57 | donnyd | will see tomorrow if it fixes the issue | |
| 20:29:48 | zigo | I got 3 unit test failures in Nova when building Stein in sid, because the tests are expecting from libvirt: | |
| 20:29:49 | zigo | <target dev="vda" bus="virtio"/> | |
| 20:29:49 | zigo | when libvirt really returns: | |
| 20:29:49 | zigo | <target bus="virtio" dev="vda"/> | |
| 20:29:55 | zigo | Has this been fixed in master? | |
| 20:30:09 | zigo | mriedem: ^ | |
| 20:32:08 | mriedem | zigo: no idea, but it's likely some test using dicts and not handling sort order on the keys | |
| 20:32:23 | zigo | Exactly. | |
| 20:32:32 | mriedem | open a bug with the test traceback | |
| 20:32:39 | zigo | Sure. | |
| 20:32:57 | zigo | mriedem: Still in Launchpad, or in Storyboard? | |
| 20:33:18 | zigo | Launchpad, it seems. | |
| 20:33:48 | sean-k-mooney | if we cold migrate after live migration it failts with a port update error | |
| 20:34:25 | sean-k-mooney | Error: Failed to perform requested operation on instance "sriov-macvtap", the instance has an error status: Please try again later [Error: Port update failed for port 485937a6-7611-4948-8420-4bfd73e15ea8: Unable to correlate PCI slot 0000:01:10.1]. | |
| 20:36:22 | zigo | mriedem: https://bugs.launchpad.net/nova/+bug/1841667 | |
| 20:36:23 | openstack | Launchpad bug 1841667 in OpenStack Compute (nova) "failing libvirt tests: need ordering" [Undecided,New] | |
| 20:36:35 | zigo | Feel free to rewrite the title if you find it not good enough. | |
| 20:38:32 | sean-k-mooney | well that is unexpected.. | |
| 20:41:09 | sean-k-mooney | it errored after seting the host to the other host e.g. when migrating form cloud-4->cloud-3 it was errord on cloud-3 but libivrt did not have a domain on cloud-3 | |
| 20:41:25 | sean-k-mooney | it still had a domian on cloud-4 but was shutoff | |
| 20:41:41 | sean-k-mooney | after a hardreboot it went form error to running on cloud-3 | |
| 20:46:51 | mriedem | zigo: what version of lxml is being used? | |
| 20:47:17 | mriedem | prometheanfire: efried: wasn't there a recent issue with newer lxml and nova unit tests? | |
| 20:47:20 | sean-k-mooney | ya i delete that after the hard reboot it was running on cloud-3 with the device claimed on cloud-4 so that is not good. | |
| 20:47:36 | sean-k-mooney | im going to unstack and call it a night. | |
| 20:47:37 | efried | mriedem: yes, sean-k-mooney was looking into it. | |
| 20:47:46 | sean-k-mooney | mriedem: ya im ment to be looking into that | |
| 20:47:49 | efried | had to do with attribute ordering. | |
| 20:47:56 | mriedem | yup https://bugs.launchpad.net/nova/+bug/1841667 | |
| 20:47:56 | sean-k-mooney | it was what i had planned to do today... | |
| 20:47:58 | openstack | Launchpad bug 1841667 in OpenStack Compute (nova) stein "failing libvirt tests: need ordering" [Undecided,New] | |
| 20:48:02 | zigo | mriedem: 4.4.1. | |
| 20:48:18 | mriedem | efried: sean-k-mooney: did we have a bug for it? | |
| 20:48:19 | zigo | That's kind of new in Sid... | |
| 20:48:29 | sean-k-mooney | there was one filed yes | |
| 20:48:40 | sean-k-mooney | mriedem: prometheanfire filed it i think | |
| 20:48:48 | openstack | Launchpad bug 1838666 in OpenStack Compute (nova) "lxml 4.4.0 causes failed tests in nova" [Undecided,Confirmed] | |
| 20:48:48 | mriedem | https://bugs.launchpad.net/nova/+bug/1838666 | |
| 20:48:50 | mriedem | got it | |
| 20:49:00 | zigo | mriedem: Uploaded 10 days ago: https://tracker.debian.org/pkg/lxml | |
| 20:51:33 | zigo | Wow, with my brand new laptop, it only takes me 5 minutes to run all Nova unit tests now! :) | |
| 20:51:51 | mriedem | sounds like a challenge | |
| 20:53:10 | sean-k-mooney | ya its nice when you can throw more cores at it and it scales well | |
| 20:53:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Raise when inventory retrieval & refresh fails https://review.opendev.org/678957 | |
| 20:53:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove @safe_connect from _get_sharing_providers https://review.opendev.org/678959 | |
| 20:53:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove @safe_connect from _get_provider_aggregates https://review.opendev.org/678958 | |
| 20:53:33 | openstackgerrit | Eric Fried proposed openstack/nova master: Invalidate cache when _refresh_associations fails https://review.opendev.org/678960 | |
| 20:53:34 | efried | mriedem: This ^ may be sufficient for bug 1841481, assuming "the next refresh" is quick enough for us to repopulate the cache. Otherwise we may wish to wrap _refresh_associations in a one-loop @retry (but not on ClientException). Let me know your thoughts if you please. | |
| 20:53:35 | openstack | bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,Triaged] https://launchpad.net/bugs/1841481 - Assigned to Eric Fried (efried) | |
| 20:55:07 | zigo | sean-k-mooney: Got a 6 core Xeon (so 12 threads...)... | |
| 20:56:08 | sean-k-mooney | avoids the whos system is faster trap. | |
| 20:59:26 | zigo | The nice thing is that I do have enough NVME space to have a full Debian mirror ... | |
| 20:59:36 | zigo | deb file:///home/ftp/debian ... :P | |
| 21:00:08 | sean-k-mooney | thats like somewhere betten 230-400GB right | |
| 21:00:33 | sean-k-mooney | i used to use apt-cacheng to cache the things i actully was using | |
| 21:00:39 | sean-k-mooney | i should set that up again. | |
| 21:00:58 | zigo | sean-k-mooney: 463 GB right now, only amd64 + sources, all available suites. | |
| 21:01:29 | zigo | The point is, it directly fetches on drive, avoiding network socket indirections... | |
| 21:01:39 | sean-k-mooney | right we had a ubunut mirror for the intel nfv ci and i rememebr it was somere in that range | |
| 21:01:44 | zigo | sean-k-mooney: I used approx. | |
| 21:01:47 | sean-k-mooney | but that was like 3 years ago | |
| 21:01:54 | sean-k-mooney | so i assuemd it grew | |
| 21:03:17 | mriedem | melwitt: time for you to bug dansmith https://review.opendev.org/#/c/507486/69 | |
| 21:03:41 | sean-k-mooney | prometheanfire: do you know if the lxml thing only happins on a specific python verions | |
| 21:03:58 | sean-k-mooney | im running the test uder py36 at the momemt | |
| 21:04:03 | mriedem | zigo: are you testing on py27 or py36? | |
| 21:04:34 | sean-k-mooney | i can test both but py36 is faster for me to test with | |
| 21:04:57 | sean-k-mooney | i have 4 test failrue on py36 | |
| 21:06:09 | sean-k-mooney | all of which are not using our xml matcher | |
| 21:06:27 | sean-k-mooney | there doing self.assertEqual(diska_xml.strip(), actual_diska_xml.strip()) | |
| 21:06:42 | sean-k-mooney | or assert called with | |
| 21:08:21 | prometheanfire | sean-k-mooney: I don't know :( | |
| 21:08:35 | sean-k-mooney | it broke on py36 so i can reporduce | |
| 21:08:40 | sean-k-mooney | so its fine | |
| 21:08:41 | zigo | mriedem: python 3.7 | |
| 21:09:12 | zigo | sean-k-mooney: py37 | |
| 21:09:23 | sean-k-mooney | it likely not python specfic | |
| 21:09:28 | zigo | Right. | |
| 21:11:08 | sean-k-mooney | actully at least one of the tests fail on py3 and pass on py27 | |
| 21:11:18 | sean-k-mooney | so maybe it is but ill fix it for both in anycase | |
| 21:14:25 | donnyd | so here is some interesting information about the image cache | |
| 21:14:37 | donnyd | I keep getting this error thrown | |
| 21:14:53 | donnyd | https://www.irccloud.com/pastebin/9eB2NYg4/ | |
| 21:15:08 | sean-k-mooney | ya i have seen that | |
| 21:15:11 | donnyd | most notably this nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d0 | |
| 21:15:32 | donnyd | however that image is there, and is continually downloaded over and over | |
| 21:15:50 | sean-k-mooney | i think mdbooth was fixing a cese were if we fail to resize an image we delete the base image backing file in the cache | |
| 21:16:20 | donnyd | https://www.irccloud.com/pastebin/yllMTx02/ | |