| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-23 | |||
| 14:10:56 | sean-k-mooney | using the ownwer_trits approch but bauzas has conserns over that as does gibi | |
| 14:11:11 | sean-k-mooney | but really this is a sperate problem form cybrog vgpu supprot | |
| 14:11:19 | bauzas | my main concern is that we need to add traits for nova hosts | |
| 14:11:42 | sean-k-mooney | yes but they are not really nova hosts | |
| 14:11:59 | bauzas | non cyborg managed hosts, if you prefer | |
| 14:12:10 | bauzas | or libvirt-managed hosts | |
| 14:12:18 | sean-k-mooney | we are supper bias but i have agrued since before nested resouce providers that nova shoudl not own the root rp | |
| 14:12:38 | sean-k-mooney | well it was one of my argument for intoducing them | |
| 14:12:55 | sean-k-mooney | the "compute" host is really a shared thing | |
| 14:13:05 | sean-k-mooney | to which multiple serivce may create resouces | |
| 14:14:07 | sean-k-mooney | being totally frank im worried that we will not come to a desicion on this this cycle and cyborg will have to with a third time to make progress | |
| 14:16:03 | sean-k-mooney | i have personaly see this type of discussion take litrally 2-3 year to progress and i think that is harmful to the openstack comunity as a whole. i also dont want to rush it as its hard to pivort after its released | |
| 14:17:32 | bauzas | we provided alternatives | |
| 14:17:58 | sean-k-mooney | yes but we dont agree on any of them | |
| 14:19:42 | sean-k-mooney | i hate to say this but i think we need a spec for this or atleast an etherpad and some midcycle like real time design session on this | |
| 14:20:04 | sean-k-mooney | this is exactly the type of thing we woule have worked though in the ptg | |
| 14:20:42 | sean-k-mooney | with a whiteboard and all the stake holders (where possible) in the same room | |
| 14:21:16 | bauzas | I can't disagree | |
| 14:21:24 | dansmith | we've also had plenty of those in the past which didn't yield much progress.. specifically about cyborg :) | |
| 14:22:05 | sean-k-mooney | ya the dublin seesion in partcalar was less then useful | |
| 14:24:06 | sean-k-mooney | gibi: thanks for trying to help | |
| 14:24:13 | sean-k-mooney | but i feel the same | |
| 14:24:36 | gibi | I summarized my lates view in the spec review but I haven't published it yet. | |
| 14:25:11 | sean-k-mooney | could we just use a custom resouce class for now an punt on this for now | |
| 14:25:22 | gibi | I'm torn between doing the right thing architecturally and supporting a parallel projec to integrate with nova technically | |
| 14:25:56 | sean-k-mooney | i know that would requrie a reshape eventually but at least it would unblcok them | |
| 14:26:11 | gibi | it would unblock them and we will never allocate time to do that reshape later | |
| 14:26:27 | sean-k-mooney | well it would be in cyborg? | |
| 14:26:29 | gibi | as it just a lot of work | |
| 14:26:36 | gibi | for basicly nothing | |
| 14:26:37 | sean-k-mooney | maybe it would be in nova | |
| 14:26:54 | sean-k-mooney | damb cross project reshapes will be a bitch to fiture out forget i said anything | |
| 14:38:54 | bauzas | gibi: thanks | |
| 14:39:09 | elodilles | sean-k-mooney gibi : as we yesterday discussed on the meeting, here is an example of a failure what I saw: test_live_block_migration_with_attached_volume -- https://c5d6a707d1df71acd55f-fbeed0693cac4a6e5441d43111515edc.ssl.cf5.rackcdn.com/787252/4/gate/nova-live-migration/706ec17/testr_results.html | |
| 14:40:17 | gibi | elodilles: looking | |
| 14:40:19 | elodilles | I saw this a couple of times at the failures of this patch: https://review.opendev.org/c/openstack/nova/+/787252 | |
| 14:40:23 | gibi | elodilles: you have a good timing :) | |
| 14:40:25 | elodilles | stable/victoria | |
| 14:40:33 | elodilles | gibi: why? o:) | |
| 14:40:41 | sean-k-mooney | weird | |
| 14:40:49 | sean-k-mooney | both thos test refernce the same volume 7688b3f1-6549-4e0c-a507-9789ddc2eb95 | |
| 14:41:08 | gibi | elodilles: I've just stopped thinking about the above vGPU problem | |
| 14:41:30 | sean-k-mooney | that should not happen right | |
| 14:41:41 | sean-k-mooney | two tests refering to the same volume uuid 7688b3f1-6549-4e0c-a507-9789ddc2eb95 | |
| 14:41:51 | sean-k-mooney | if they run in parally they would conflict | |
| 14:42:00 | gibi | sean-k-mooney: wich two test cases? | |
| 14:42:05 | gibi | sean-k-mooney: I see a testcase and a tear down | |
| 14:42:11 | gibi | of the suite | |
| 14:42:19 | sean-k-mooney | oh ya its the tear down method | |
| 14:42:29 | sean-k-mooney | ok that make a liit more sense | |
| 14:42:29 | elodilles | gibi: well I've seen that there was an ongoing discussion so I did not want to interrupt earlier :) | |
| 14:43:42 | sean-k-mooney | ok so the teardown failed becasue presumable the volume was still attached to the vm | |
| 14:43:54 | sean-k-mooney | it was in state detaching | |
| 14:46:28 | gibi | it is trying to detach in a loop so I guess it is the original detach problem | |
| 14:46:39 | gibi | for what we created the notification based solution | |
| 14:46:47 | gibi | but that solution is not in victoria | |
| 14:46:58 | sean-k-mooney | ya the detach starts at Jun 21 09:20:16.007823 | |
| 14:48:00 | gibi | so either we increase the timeout value before we retry in victoria or backport the notification based solution to victoria | |
| 14:48:39 | sean-k-mooney | we are calling os_brick.initiator.connectors.iscsi.ISCSIConnector.disconnect_volume on it | |
| 14:48:59 | sean-k-mooney | we would not do that if the detach ahs not commpeted right? | |
| 14:49:37 | gibi | sean-k-mooney: I don't know. what I know that the detach is not completed from libvirt perspective, it still reports the device in the live domain after 7 retries | |
| 14:50:03 | sean-k-mooney | this is happening as part of a live migration | |
| 14:51:08 | sean-k-mooney | this looks odd what is the test actully doing | |
| 14:51:38 | sean-k-mooney | it look like we are runign post live migration on the souce which is tearing odwn the volume while its potentioanly still detaching | |
| 14:51:58 | gibi | I think lyarwood has more context on this detach problem | |
| 14:52:13 | sean-k-mooney | im wonderign if the tempest test is broken | |
| 14:52:42 | sean-k-mooney | e.g. is it issuing a detach and then a live migrate without waiting for the detach to complete | |
| 14:53:45 | sean-k-mooney | this is the test https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_live_migration.py#L178-L204 | |
| 14:54:30 | sean-k-mooney | thats doing an attach | |
| 14:54:41 | sean-k-mooney | but its not doing a detach | |
| 14:54:53 | sean-k-mooney | its also not waiting to ensure its actully attached | |
| 14:55:06 | sean-k-mooney | unless self.attach_volume is doint that internally | |
| 14:56:31 | sean-k-mooney | ok if the attach fails its cleanup funciton does a detach im assuming https://github.com/openstack/tempest/blob/53c02181f87804a4ba8ddf6288ea1f7717234c2a/tempest/api/compute/base.py#L554-L591 | |
| 14:56:47 | sean-k-mooney | it is waiting internally | |
| 14:57:42 | sean-k-mooney | https://github.com/openstack/tempest/blob/53c02181f87804a4ba8ddf6288ea1f7717234c2a/tempest/api/compute/base.py#L590 | |
| 14:58:02 | sean-k-mooney | can we see the volume attach complete in the logs | |
| 15:00:39 | sean-k-mooney | we do the attachment in libvirt at un 21 09:20:00.476733 | |
| 15:10:10 | lyarwood | if we are still talking about https://c5d6a707d1df71acd55f-fbeed0693cac4a6e5441d43111515edc.ssl.cf5.rackcdn.com/787252/4/gate/nova-live-migration/706ec17/testr_results.html it's just another basic detach failure from the live domain | |
| 15:10:22 | lyarwood | after the test has passed and we are cleaning up | |
| 15:12:12 | lyarwood | I really need to land https://review.opendev.org/c/openstack/tempest/+/794757 so we can get some guestOS logs in this case | |
| 15:13:30 | gibi | lyarwood: that run is from stable/victoria so we don't have the new detach code there. Do you think it would help if we try to backport the new detach code? or is it possibly unrelated? | |
| 15:13:31 | lyarwood | or we could just Depends-On that from a DNM stable/victoria change | |
| 15:13:38 | elodilles | lyarwood: so you are saying that this is similar like bug #1931716 but not the exact same case? | |
| 15:14:04 | lyarwood | gibi: we've seen failures of this test on master so this time I don't think backporting that code is going to help | |
| 15:14:06 | lyarwood | brb | |
| 15:15:15 | gibi | lyarwood: OK, then I agree to land the tempest improvement first to get more info | |
| 15:18:44 | opendevreview | Elod Illes proposed openstack/nova stable/victoria: DNM: volume detach test https://review.opendev.org/c/openstack/nova/+/797675 | |
| 15:19:30 | elodilles | I've created the DNM patch what lyarwood suggested ^^^ | |
| 15:21:51 | lyarwood | elodilles: thanks | |
| 15:22:23 | elodilles | np. let's see what happens :) | |
| 15:30:06 | lyarwood | elodilles: I have a change to skip the test on master btw that we could also backport if we see the same soft lockups | |
| 15:30:42 | opendevreview | Stephen Finucane proposed openstack/nova stable/victoria: Test numa and vcpu topologies bug: #1910466 https://review.opendev.org/c/openstack/nova/+/797680 | |
| 15:30:43 | opendevreview | Stephen Finucane proposed openstack/nova stable/victoria: Fix max cpu topologies with numa affinity https://review.opendev.org/c/openstack/nova/+/797681 | |
| 15:35:54 | elodilles | lyarwood: thanks, good to know that, we might need to backport that then if needed | |
| 15:46:02 | kashyap | gibi: lyarwood: sean-k-mooney: When you get a min, have a look to see if you spot any holes there: https://blueprints.launchpad.net/nova/+spec/virtio-as-default-display-device | |
| 15:50:50 | sean-k-mooney | that will need a spec most likely | |
| 15:51:11 | sean-k-mooney | im not agaisnt it but we need to record the current modele in use | |
| 15:51:17 | sean-k-mooney | and then only change the default for new instnaces | |
| 15:59:48 | sean-k-mooney | lyarwood: so in this case we dont actully want to do the detach on succesful migration right | |
| 16:00:01 | sean-k-mooney | that is jsut an artifcat of the way the cleanup is done | |