| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 20:02:31 | sean-k-mooney | commeted and -1'd | |
| 20:06:21 | artom | Sheesh, rub it in, will'ya | |
| 20:07:03 | mriedem | i can run a -2 into it if that helps | |
| 20:07:06 | mriedem | *rub | |
| 20:07:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update help for image_cache_manager_interval option https://review.opendev.org/678954 | |
| 20:07:10 | sean-k-mooney | rubbing salt in a wound helps it heal :P | |
| 20:07:26 | artom | mriedem, no, rubbing a -2 does something entirely different | |
| 20:12:18 | sean-k-mooney | in other new https://bugs.launchpad.net/nova/+bug/1804502 is a pain | |
| 20:12:19 | openstack | Launchpad bug 1804502 in OpenStack Compute (nova) "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Undecided,In progress] - Assigned to David Hill (david-hill-ubisoft) | |
| 20:12:55 | sean-k-mooney | and yes i know its a feature not a bug but i think i need to go figure out how to make that work for U ... | |
| 20:12:58 | mriedem | cfriesen has a duplicate of that i think | |
| 20:13:12 | sean-k-mooney | ya its a long knonw issue | |
| 20:13:26 | sean-k-mooney | none of the filters are really aware of rebuild/resize | |
| 20:13:39 | sean-k-mooney | and they all reate the request as a new spawn | |
| 20:13:56 | sean-k-mooney | so we need double the capasity yada yada yada | |
| 20:14:12 | sean-k-mooney | its fixable it just a pain | |
| 20:14:38 | sean-k-mooney | and we have had more important things like moveing this stuff to placment | |
| 20:14:43 | mriedem | the filters are definitely aware of rebuild now | |
| 20:14:55 | sean-k-mooney | not all of them | |
| 20:15:05 | sean-k-mooney | or rahter they have to be coded to be aware | |
| 20:15:19 | mriedem | correct, https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L26 | |
| 20:15:26 | sean-k-mooney | we now at least go back to the schduler when the image changes | |
| 20:15:42 | dansmith | artom: even if there's technically nothing that breaks because we nuke pci_requests, I still don't think it makes sense to land it in a state where we're clobbering data, which I think is what you're getting at | |
| 20:16:00 | sean-k-mooney | yes it needs to run on rebuild but it does not have special handeling for rebuild | |
| 20:17:20 | mriedem | dansmith: i woudln't be surprised if a secondary operatoin on the instance after the clobbered instance live migratoin blows up | |
| 20:17:34 | dansmith | mriedem: right, I said that earlier | |
| 20:17:34 | mriedem | have seen all kinds of fun stuff with things like that and accidentally persisting crap to the request spec | |
| 20:17:40 | dansmith | mriedem: I just didn't go looking for it | |
| 20:17:42 | sean-k-mooney | i can test that if ye like | |
| 20:17:48 | sean-k-mooney | i still have it | |
| 20:17:55 | dansmith | even if we can't poke something, it's still not okay to clobber them I think | |
| 20:18:00 | sean-k-mooney | but we obviously should still fix it | |
| 20:18:08 | dansmith | anything on the compute or cell conductor that needs pci requests wouldn't be able to get it from the reqspec | |
| 20:18:27 | dansmith | and as it is with pci, it's all very untested, so it'd be easy to miss something we're breaking | |
| 20:18:29 | sean-k-mooney | yes that is true | |
| 20:19:52 | mriedem | i'm not even sure what you'd test after the clobbered live migration to show it fails, or how we'd recreate that in a functional test to prevent future regressions, but i know that having sean manually test all these weird things isn't sustainable | |
| 20:20:22 | dansmith | obviously | |
| 20:20:25 | mriedem | i guess you could like cold migrate back to the source host and verify there are any claimed pci devices for the instance | |
| 20:21:03 | sean-k-mooney | the pci device are still tracked in the pci tacker as used by the instace | |
| 20:21:11 | mriedem | oh great | |
| 20:21:18 | sean-k-mooney | actully i should check that | |
| 20:21:19 | mriedem | didn't we just talk about something like this...vpmem? | |
| 20:21:49 | sean-k-mooney | kind of | |
| 20:23:41 | sean-k-mooney | yes just check we are correctly tracking thing at the host level in the pci tracker at least | |
| 20:25:35 | donnyd | mriedem: melwitt I have the image cache dir all setup and running | |
| 20:25:57 | donnyd | will see tomorrow if it fixes the issue | |
| 20:29:48 | zigo | I got 3 unit test failures in Nova when building Stein in sid, because the tests are expecting from libvirt: | |
| 20:29:49 | zigo | <target bus="virtio" dev="vda"/> | |
| 20:29:49 | zigo | when libvirt really returns: | |
| 20:29:49 | zigo | <target dev="vda" bus="virtio"/> | |
| 20:29:55 | zigo | Has this been fixed in master? | |
| 20:30:09 | zigo | mriedem: ^ | |
| 20:32:08 | mriedem | zigo: no idea, but it's likely some test using dicts and not handling sort order on the keys | |
| 20:32:23 | zigo | Exactly. | |
| 20:32:32 | mriedem | open a bug with the test traceback | |
| 20:32:39 | zigo | Sure. | |
| 20:32:57 | zigo | mriedem: Still in Launchpad, or in Storyboard? | |
| 20:33:18 | zigo | Launchpad, it seems. | |
| 20:33:48 | sean-k-mooney | if we cold migrate after live migration it failts with a port update error | |
| 20:34:25 | sean-k-mooney | Error: Failed to perform requested operation on instance "sriov-macvtap", the instance has an error status: Please try again later [Error: Port update failed for port 485937a6-7611-4948-8420-4bfd73e15ea8: Unable to correlate PCI slot 0000:01:10.1]. | |
| 20:36:22 | zigo | mriedem: https://bugs.launchpad.net/nova/+bug/1841667 | |
| 20:36:23 | openstack | Launchpad bug 1841667 in OpenStack Compute (nova) "failing libvirt tests: need ordering" [Undecided,New] | |
| 20:36:35 | zigo | Feel free to rewrite the title if you find it not good enough. | |
| 20:38:32 | sean-k-mooney | well that is unexpected.. | |
| 20:41:09 | sean-k-mooney | it errored after seting the host to the other host e.g. when migrating form cloud-4->cloud-3 it was errord on cloud-3 but libivrt did not have a domain on cloud-3 | |
| 20:41:25 | sean-k-mooney | it still had a domian on cloud-4 but was shutoff | |
| 20:41:41 | sean-k-mooney | after a hardreboot it went form error to running on cloud-3 | |
| 20:46:51 | mriedem | zigo: what version of lxml is being used? | |
| 20:47:17 | mriedem | prometheanfire: efried: wasn't there a recent issue with newer lxml and nova unit tests? | |
| 20:47:20 | sean-k-mooney | ya i delete that after the hard reboot it was running on cloud-3 with the device claimed on cloud-4 so that is not good. | |
| 20:47:36 | sean-k-mooney | im going to unstack and call it a night. | |
| 20:47:37 | efried | mriedem: yes, sean-k-mooney was looking into it. | |
| 20:47:46 | sean-k-mooney | mriedem: ya im ment to be looking into that | |
| 20:47:49 | efried | had to do with attribute ordering. | |
| 20:47:56 | sean-k-mooney | it was what i had planned to do today... | |
| 20:47:56 | mriedem | yup https://bugs.launchpad.net/nova/+bug/1841667 | |
| 20:47:58 | openstack | Launchpad bug 1841667 in OpenStack Compute (nova) stein "failing libvirt tests: need ordering" [Undecided,New] | |
| 20:48:02 | zigo | mriedem: 4.4.1. | |
| 20:48:18 | mriedem | efried: sean-k-mooney: did we have a bug for it? | |
| 20:48:19 | zigo | That's kind of new in Sid... | |
| 20:48:29 | sean-k-mooney | there was one filed yes | |
| 20:48:40 | sean-k-mooney | mriedem: prometheanfire filed it i think | |
| 20:48:48 | mriedem | https://bugs.launchpad.net/nova/+bug/1838666 | |
| 20:48:48 | openstack | Launchpad bug 1838666 in OpenStack Compute (nova) "lxml 4.4.0 causes failed tests in nova" [Undecided,Confirmed] | |
| 20:48:50 | mriedem | got it | |
| 20:49:00 | zigo | mriedem: Uploaded 10 days ago: https://tracker.debian.org/pkg/lxml | |
| 20:51:33 | zigo | Wow, with my brand new laptop, it only takes me 5 minutes to run all Nova unit tests now! :) | |
| 20:51:51 | mriedem | sounds like a challenge | |
| 20:53:10 | sean-k-mooney | ya its nice when you can throw more cores at it and it scales well | |
| 20:53:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Raise when inventory retrieval & refresh fails https://review.opendev.org/678957 | |
| 20:53:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove @safe_connect from _get_provider_aggregates https://review.opendev.org/678958 | |
| 20:53:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove @safe_connect from _get_sharing_providers https://review.opendev.org/678959 | |
| 20:53:33 | openstackgerrit | Eric Fried proposed openstack/nova master: Invalidate cache when _refresh_associations fails https://review.opendev.org/678960 | |
| 20:53:34 | efried | mriedem: This ^ may be sufficient for bug 1841481, assuming "the next refresh" is quick enough for us to repopulate the cache. Otherwise we may wish to wrap _refresh_associations in a one-loop @retry (but not on ClientException). Let me know your thoughts if you please. | |
| 20:53:35 | openstack | bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,Triaged] https://launchpad.net/bugs/1841481 - Assigned to Eric Fried (efried) | |
| 20:55:07 | zigo | sean-k-mooney: Got a 6 core Xeon (so 12 threads...)... | |
| 20:56:08 | sean-k-mooney | avoids the whos system is faster trap. | |
| 20:59:26 | zigo | The nice thing is that I do have enough NVME space to have a full Debian mirror ... | |
| 20:59:36 | zigo | deb file:///home/ftp/debian ... :P | |
| 21:00:08 | sean-k-mooney | thats like somewhere betten 230-400GB right | |