Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-27
19:36:35 artom I need to drop for a bit while I reconnect to phone's hotspot
19:36:39 artom brb
19:37:34 sean-k-mooney well im not sure i qualify in that regard but ill test it none the less
19:42:29 sean-k-mooney the migration context is here http://paste.openstack.org/show/765743/
19:42:59 sean-k-mooney and the pci request in the instance extra table are empty
19:43:03 sean-k-mooney mysql> select pci_requests from instance_extra as ie where ie.instance_uuid='b304fedf-aa31-4990-9a74-29729eed336d';
19:43:05 sean-k-mooney +--------------+
19:43:07 sean-k-mooney | pci_requests |
19:43:09 sean-k-mooney +--------------+
19:43:11 sean-k-mooney | [] |
19:43:13 sean-k-mooney +--------------+
19:44:28 artom Yey :(
19:44:36 artom At least we know we weren't wrong to worry
19:45:11 sean-k-mooney the pci device will will be claimed with the instance uuid in the pci tracker
19:45:28 sean-k-mooney but its not correct.
19:45:31 artom Yeah, but won't that screw up future operations?
19:45:47 artom Suddenly you can evacuate to a host with no SRIOV, for example
19:46:07 sean-k-mooney i would have to do more testing but non nessiarally
19:47:15 artom Still seems dangerous
19:47:36 artom I mean, I want NUMA LM to land a *lot*, but there's a minimum level of quality here ;)
19:47:44 artom And that's not part of it
19:49:09 sean-k-mooney ya i know. the request_spec still has the orginal pci_request in it and we do not support pci hot attach
19:49:37 sean-k-mooney that is the request spec http://paste.openstack.org/show/765824/
19:49:38 openstackgerrit Matt Riedemann proposed openstack/nova master: FUP for I66d8f06f19c5c631e33208580428aa843abb38d2 https://review.opendev.org/678951
19:52:29 sean-k-mooney stephenfin: by the way i think we need to apply https://review.opendev.org/#/c/675776/
19:52:40 sean-k-mooney i am seeing some other issue that that will likely fix
19:54:50 sean-k-mooney artom: with the same code if i use a non numa flavor the pci reuests are not removed
19:55:17 sean-k-mooney artom: we might be able to use the temporay mutation thing to ensure that the change is not saved
19:56:31 artom sean-k-mooney, errr...
19:56:37 artom We want to save the *other* stuff
19:59:05 mriedem sean-k-mooney: so definite -1 here right? https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@307
19:59:10 mriedem can someone do that?
19:59:42 sean-k-mooney mriedem: yes
20:02:31 sean-k-mooney commeted and -1'd
20:06:21 artom Sheesh, rub it in, will'ya
20:07:03 mriedem i can run a -2 into it if that helps
20:07:06 mriedem *rub
20:07:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Update help for image_cache_manager_interval option https://review.opendev.org/678954
20:07:10 sean-k-mooney rubbing salt in a wound helps it heal :P
20:07:26 artom mriedem, no, rubbing a -2 does something entirely different
20:12:18 sean-k-mooney in other new https://bugs.launchpad.net/nova/+bug/1804502 is a pain
20:12:19 openstack Launchpad bug 1804502 in OpenStack Compute (nova) "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Undecided,In progress] - Assigned to David Hill (david-hill-ubisoft)
20:12:55 sean-k-mooney and yes i know its a feature not a bug but i think i need to go figure out how to make that work for U ...
20:12:58 mriedem cfriesen has a duplicate of that i think
20:13:12 sean-k-mooney ya its a long knonw issue
20:13:26 sean-k-mooney none of the filters are really aware of rebuild/resize
20:13:39 sean-k-mooney and they all reate the request as a new spawn
20:13:56 sean-k-mooney so we need double the capasity yada yada yada
20:14:12 sean-k-mooney its fixable it just a pain
20:14:38 sean-k-mooney and we have had more important things like moveing this stuff to placment
20:14:43 mriedem the filters are definitely aware of rebuild now
20:14:55 sean-k-mooney not all of them
20:15:05 sean-k-mooney or rahter they have to be coded to be aware
20:15:19 mriedem correct, https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L26
20:15:26 sean-k-mooney we now at least go back to the schduler when the image changes
20:15:42 dansmith artom: even if there's technically nothing that breaks because we nuke pci_requests, I still don't think it makes sense to land it in a state where we're clobbering data, which I think is what you're getting at
20:16:00 sean-k-mooney yes it needs to run on rebuild but it does not have special handeling for rebuild
20:17:20 mriedem dansmith: i woudln't be surprised if a secondary operatoin on the instance after the clobbered instance live migratoin blows up
20:17:34 mriedem have seen all kinds of fun stuff with things like that and accidentally persisting crap to the request spec
20:17:34 dansmith mriedem: right, I said that earlier
20:17:40 dansmith mriedem: I just didn't go looking for it
20:17:42 sean-k-mooney i can test that if ye like
20:17:48 sean-k-mooney i still have it
20:17:55 dansmith even if we can't poke something, it's still not okay to clobber them I think
20:18:00 sean-k-mooney but we obviously should still fix it
20:18:08 dansmith anything on the compute or cell conductor that needs pci requests wouldn't be able to get it from the reqspec
20:18:27 dansmith and as it is with pci, it's all very untested, so it'd be easy to miss something we're breaking
20:18:29 sean-k-mooney yes that is true
20:19:52 mriedem i'm not even sure what you'd test after the clobbered live migration to show it fails, or how we'd recreate that in a functional test to prevent future regressions, but i know that having sean manually test all these weird things isn't sustainable
20:20:22 dansmith obviously
20:20:25 mriedem i guess you could like cold migrate back to the source host and verify there are any claimed pci devices for the instance
20:21:03 sean-k-mooney the pci device are still tracked in the pci tacker as used by the instace
20:21:11 mriedem oh great
20:21:18 sean-k-mooney actully i should check that
20:21:19 mriedem didn't we just talk about something like this...vpmem?
20:21:49 sean-k-mooney kind of
20:23:41 sean-k-mooney yes just check we are correctly tracking thing at the host level in the pci tracker at least
20:25:35 donnyd mriedem: melwitt I have the image cache dir all setup and running
20:25:57 donnyd will see tomorrow if it fixes the issue
20:29:48 zigo I got 3 unit test failures in Nova when building Stein in sid, because the tests are expecting from libvirt:
20:29:49 zigo <target dev="vda" bus="virtio"/>
20:29:49 zigo when libvirt really returns:
20:29:49 zigo <target bus="virtio" dev="vda"/>
20:29:55 zigo Has this been fixed in master?
20:30:09 zigo mriedem: ^
20:32:08 mriedem zigo: no idea, but it's likely some test using dicts and not handling sort order on the keys
20:32:23 zigo Exactly.
20:32:32 mriedem open a bug with the test traceback
20:32:39 zigo Sure.
20:32:57 zigo mriedem: Still in Launchpad, or in Storyboard?
20:33:18 zigo Launchpad, it seems.
20:33:48 sean-k-mooney if we cold migrate after live migration it failts with a port update error
20:34:25 sean-k-mooney Error: Failed to perform requested operation on instance "sriov-macvtap", the instance has an error status: Please try again later [Error: Port update failed for port 485937a6-7611-4948-8420-4bfd73e15ea8: Unable to correlate PCI slot 0000:01:10.1].
20:36:22 zigo mriedem: https://bugs.launchpad.net/nova/+bug/1841667
20:36:23 openstack Launchpad bug 1841667 in OpenStack Compute (nova) "failing libvirt tests: need ordering" [Undecided,New]
20:36:35 zigo Feel free to rewrite the title if you find it not good enough.
20:38:32 sean-k-mooney well that is unexpected..
20:41:09 sean-k-mooney it errored after seting the host to the other host e.g. when migrating form cloud-4->cloud-3 it was errord on cloud-3 but libivrt did not have a domain on cloud-3
20:41:25 sean-k-mooney it still had a domian on cloud-4 but was shutoff
20:41:41 sean-k-mooney after a hardreboot it went form error to running on cloud-3

Earlier   Later