| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 19:30:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.opendev.org/642183 | |
| 19:30:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change HostManager to allow scheduling to other cells https://review.opendev.org/614037 | |
| 19:30:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Destination.allow_cross_cell_move field https://review.opendev.org/614035 | |
| 19:30:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.opendev.org/633293 | |
| 19:30:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832 | |
| 19:30:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.opendev.org/627890 | |
| 19:30:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080 | |
| 19:30:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891 | |
| 19:30:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | |
| 19:30:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646 | |
| 19:30:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | |
| 19:30:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684 | |
| 19:30:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | |
| 19:30:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | |
| 19:30:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | |
| 19:30:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | |
| 19:31:12 | mriedem | gibi: stephenfin: dansmith: ^ i've rebased and moved the already +2'ed and less controversial things to the bottom to help move those through | |
| 19:32:24 | sean-k-mooney | artom: http://paste.openstack.org/show/765671/ | |
| 19:32:30 | donnyd | mriedem: you are supposed to wait for Friday afternoons for that kind of action.. so my CI has something to do... it gets lonely and bored | |
| 19:32:37 | sean-k-mooney | that the xml before and after the migration | |
| 19:33:14 | mriedem | donnyd: i wanted to test your CI's image download capabilities | |
| 19:33:29 | donnyd | LOL - thanks | |
| 19:35:05 | artom | sean-k-mooney, I think the best thing would be to write up a comprehensive test report | |
| 19:35:30 | artom | I'm fairly confident that the XML part works, that's been there since Stein | |
| 19:35:49 | artom | It's more the new surprises - pci_requests from the migration_context, etc | |
| 19:36:04 | artom | It doesn't help that the path series keeps changing under you | |
| 19:36:05 | artom | But yeah | |
| 19:36:20 | artom | I think testing by a more or less independent third party would be awesome :) | |
| 19:36:35 | artom | I need to drop for a bit while I reconnect to phone's hotspot | |
| 19:36:39 | artom | brb | |
| 19:37:34 | sean-k-mooney | well im not sure i qualify in that regard but ill test it none the less | |
| 19:42:29 | sean-k-mooney | the migration context is here http://paste.openstack.org/show/765743/ | |
| 19:42:59 | sean-k-mooney | and the pci request in the instance extra table are empty | |
| 19:43:03 | sean-k-mooney | mysql> select pci_requests from instance_extra as ie where ie.instance_uuid='b304fedf-aa31-4990-9a74-29729eed336d'; | |
| 19:43:05 | sean-k-mooney | +--------------+ | |
| 19:43:07 | sean-k-mooney | | pci_requests | | |
| 19:43:09 | sean-k-mooney | +--------------+ | |
| 19:43:11 | sean-k-mooney | | [] | | |
| 19:43:13 | sean-k-mooney | +--------------+ | |
| 19:44:28 | artom | Yey :( | |
| 19:44:36 | artom | At least we know we weren't wrong to worry | |
| 19:45:11 | sean-k-mooney | the pci device will will be claimed with the instance uuid in the pci tracker | |
| 19:45:28 | sean-k-mooney | but its not correct. | |
| 19:45:31 | artom | Yeah, but won't that screw up future operations? | |
| 19:45:47 | artom | Suddenly you can evacuate to a host with no SRIOV, for example | |
| 19:46:07 | sean-k-mooney | i would have to do more testing but non nessiarally | |
| 19:47:15 | artom | Still seems dangerous | |
| 19:47:36 | artom | I mean, I want NUMA LM to land a *lot*, but there's a minimum level of quality here ;) | |
| 19:47:44 | artom | And that's not part of it | |
| 19:49:09 | sean-k-mooney | ya i know. the request_spec still has the orginal pci_request in it and we do not support pci hot attach | |
| 19:49:37 | sean-k-mooney | that is the request spec http://paste.openstack.org/show/765824/ | |
| 19:49:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: FUP for I66d8f06f19c5c631e33208580428aa843abb38d2 https://review.opendev.org/678951 | |
| 19:52:29 | sean-k-mooney | stephenfin: by the way i think we need to apply https://review.opendev.org/#/c/675776/ | |
| 19:52:40 | sean-k-mooney | i am seeing some other issue that that will likely fix | |
| 19:54:50 | sean-k-mooney | artom: with the same code if i use a non numa flavor the pci reuests are not removed | |
| 19:55:17 | sean-k-mooney | artom: we might be able to use the temporay mutation thing to ensure that the change is not saved | |
| 19:56:31 | artom | sean-k-mooney, errr... | |
| 19:56:37 | artom | We want to save the *other* stuff | |
| 19:59:05 | mriedem | sean-k-mooney: so definite -1 here right? https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@307 | |
| 19:59:10 | mriedem | can someone do that? | |
| 19:59:42 | sean-k-mooney | mriedem: yes | |
| 20:02:31 | sean-k-mooney | commeted and -1'd | |
| 20:06:21 | artom | Sheesh, rub it in, will'ya | |
| 20:07:03 | mriedem | i can run a -2 into it if that helps | |
| 20:07:06 | mriedem | *rub | |
| 20:07:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update help for image_cache_manager_interval option https://review.opendev.org/678954 | |
| 20:07:10 | sean-k-mooney | rubbing salt in a wound helps it heal :P | |
| 20:07:26 | artom | mriedem, no, rubbing a -2 does something entirely different | |
| 20:12:18 | sean-k-mooney | in other new https://bugs.launchpad.net/nova/+bug/1804502 is a pain | |
| 20:12:19 | openstack | Launchpad bug 1804502 in OpenStack Compute (nova) "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Undecided,In progress] - Assigned to David Hill (david-hill-ubisoft) | |
| 20:12:55 | sean-k-mooney | and yes i know its a feature not a bug but i think i need to go figure out how to make that work for U ... | |
| 20:12:58 | mriedem | cfriesen has a duplicate of that i think | |
| 20:13:12 | sean-k-mooney | ya its a long knonw issue | |
| 20:13:26 | sean-k-mooney | none of the filters are really aware of rebuild/resize | |
| 20:13:39 | sean-k-mooney | and they all reate the request as a new spawn | |
| 20:13:56 | sean-k-mooney | so we need double the capasity yada yada yada | |
| 20:14:12 | sean-k-mooney | its fixable it just a pain | |
| 20:14:38 | sean-k-mooney | and we have had more important things like moveing this stuff to placment | |
| 20:14:43 | mriedem | the filters are definitely aware of rebuild now | |
| 20:14:55 | sean-k-mooney | not all of them | |
| 20:15:05 | sean-k-mooney | or rahter they have to be coded to be aware | |
| 20:15:19 | mriedem | correct, https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L26 | |
| 20:15:26 | sean-k-mooney | we now at least go back to the schduler when the image changes | |
| 20:15:42 | dansmith | artom: even if there's technically nothing that breaks because we nuke pci_requests, I still don't think it makes sense to land it in a state where we're clobbering data, which I think is what you're getting at | |
| 20:16:00 | sean-k-mooney | yes it needs to run on rebuild but it does not have special handeling for rebuild | |
| 20:17:20 | mriedem | dansmith: i woudln't be surprised if a secondary operatoin on the instance after the clobbered instance live migratoin blows up | |
| 20:17:34 | mriedem | have seen all kinds of fun stuff with things like that and accidentally persisting crap to the request spec | |
| 20:17:34 | dansmith | mriedem: right, I said that earlier | |
| 20:17:40 | dansmith | mriedem: I just didn't go looking for it | |
| 20:17:42 | sean-k-mooney | i can test that if ye like | |
| 20:17:48 | sean-k-mooney | i still have it | |
| 20:17:55 | dansmith | even if we can't poke something, it's still not okay to clobber them I think | |
| 20:18:00 | sean-k-mooney | but we obviously should still fix it | |
| 20:18:08 | dansmith | anything on the compute or cell conductor that needs pci requests wouldn't be able to get it from the reqspec | |
| 20:18:27 | dansmith | and as it is with pci, it's all very untested, so it'd be easy to miss something we're breaking | |
| 20:18:29 | sean-k-mooney | yes that is true | |
| 20:19:52 | mriedem | i'm not even sure what you'd test after the clobbered live migration to show it fails, or how we'd recreate that in a functional test to prevent future regressions, but i know that having sean manually test all these weird things isn't sustainable | |
| 20:20:22 | dansmith | obviously | |
| 20:20:25 | mriedem | i guess you could like cold migrate back to the source host and verify there are any claimed pci devices for the instance | |
| 20:21:03 | sean-k-mooney | the pci device are still tracked in the pci tacker as used by the instace | |