| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-10 | |||
| 13:14:44 | stephenfin | efried: I am, and I'm going to hold off on the follow-up requested for https://review.opendev.org/#/c/674895/ because reviews seem more important at the moment, right? | |
| 13:14:45 | efried | brinzhang: looks like you already got your novaclient release merging, yah? | |
| 13:15:10 | efried | stephenfin: totally | |
| 13:16:36 | openstackgerrit | Merged openstack/nova master: Indent fake libvirt host capabilities fixtures more nicely https://review.opendev.org/679339 | |
| 13:16:37 | openstack | Launchpad bug 1843433 in OpenStack Compute (nova) "functional test test_migrate_server_with_qos_port fails intermittently due to race condition" [Undecided,New] | |
| 13:16:37 | gibi | mriedem: field a bug https://bugs.launchpad.net/nova/+bug/1843433 | |
| 13:17:22 | bauzas | gibi: you could triage it as critical since it impacts the gate IMHO | |
| 13:17:35 | bauzas | gibi: any ideas of the failure rate? | |
| 13:17:37 | gibi | bauzas: does it block the gate? | |
| 13:17:48 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22in%20test_migrate_server_with_qos_port%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 13:17:51 | mriedem | it's not critical | |
| 13:17:55 | mriedem | yes it hits the gate but doesn't block it | |
| 13:18:04 | bauzas | mriedem: okay, thanks, I was about to ask logstash | |
| 13:18:05 | mriedem | gibi: ack, i left a few comments in the patch to close the race | |
| 13:18:16 | gibi | mriedem: thanks. I will respin soon | |
| 13:18:21 | bauzas | gibi: what's the patch up for this ? | |
| 13:18:34 | gibi | bauzas: https://review.opendev.org/#/c/681238/1 | |
| 13:19:00 | bauzas | gibi: tbh, I'm a bit afraid it's at the top of the series | |
| 13:19:16 | bauzas | if you need to respin, I'd appreciate you move it down | |
| 13:19:25 | gibi | bauzas: I think it is top of master | |
| 13:19:25 | luyao | efried: Could you help confirm will an rebuild instance have two groups of allocations? I knew migration instance will change one allocation consumer to migration uuid. https://review.opendev.org/#/c/678452/23/nova/compute/resource_tracker.py@406 | |
| 13:19:32 | bauzas | yup | |
| 13:19:49 | gibi | bauzas: so it is not connected to the series | |
| 13:19:57 | gibi | bauzas: just by topic | |
| 13:20:08 | mriedem | right, it's a race in an already merged functional test, | |
| 13:20:13 | mriedem | so let's just fix the race and merge it | |
| 13:20:21 | bauzas | gibi: my bad, you're right | |
| 13:20:26 | gibi | mriedem: good suggestion about instance event, that will fix the race | |
| 13:20:33 | mriedem | \o/ | |
| 13:20:56 | gibi | I was affraid we need to poll the migration allocation in placement to close the race | |
| 13:21:19 | gibi | mriedem: I also fixed the bug grenade found in revert | |
| 13:21:40 | gibi | mriedem: unfortunately we still not have test run on that patch | |
| 13:21:53 | aspiers | kashyap: oh, I thought you were talking about cpuid inside the guest | |
| 13:22:09 | mriedem | gibi: yeah i saw you updated but i haven't looked at your replies or changes, | |
| 13:22:13 | aspiers | kashyap: I think there are multiple binaries with SB | |
| 13:22:17 | mriedem | it kind of sucks that we don't store the information needed for revert in the migration context | |
| 13:22:41 | gibi | mriedem: I have better plans, described in my reply | |
| 13:22:56 | gibi | mriedem: we can let neutron remember the mapping | |
| 13:23:09 | kashyap | aspiers: Sorry, I was not clear. I don't _know_ if it is also reported in the guest `cpuid` -- maybe you can tell from your hardware? | |
| 13:23:10 | gibi | mriedem: by using multiple portbinding | |
| 13:23:27 | aspiers | kashyap: I don't have a guest with cpuid installed currently | |
| 13:23:46 | aspiers | kashyap: I think I misremembered - before I probably ran it on the host | |
| 13:24:19 | kashyap | aspiers: For SB -- IMHO, you don't _need_ all those 4M, 2M variants -- it's beyond overkill. Your life, and admins life will be far more simpler, if you used the "two pairs" approach I noted in the comment | |
| 13:24:43 | aspiers | kashyap: You're preaching to the choir. You really need to file a bug report on bugzilla.opensuse.org | |
| 13:24:47 | mriedem | gibi: ah yeah | |
| 13:24:53 | sean-k-mooney | artom: i have confirmed that the migration context correctly contains the new numa toplogy blob but it does not get saved to the db | |
| 13:25:02 | aspiers | kashyap: or reach the right team in some other way | |
| 13:25:07 | kashyap | aspiers: Aaah, sorry; didn't realize we're on the same line, same word | |
| 13:25:23 | openstackgerrit | Adam Spiers proposed openstack/nova master: Improve SEV documentation and other minor tweaks https://review.opendev.org/681254 | |
| 13:25:50 | aspiers | efried, gibi, stephenfin: there's the follow-up ^^^ | |
| 13:25:58 | sean-k-mooney | artom: so do ing apply migration context followed by drop to delete it form the db does not work | |
| 13:26:23 | sean-k-mooney | artom: i would assume that for some reason the filed is not flaged as dirty and is not being saved | |
| 13:26:37 | gibi | aspiers: ack! | |
| 13:27:59 | Sundar | Hi sean-k-mooney, how are you doing? | |
| 13:28:07 | sean-k-mooney | Sundar: hi | |
| 13:28:11 | efried | luyao: I'm trying to make sense of those scenarios right now. | |
| 13:28:34 | Sundar | sean-k-mooney: The Cyborg patches for nova-integ have merged; https://review.opendev.org/#/q/project:openstack/cyborg+branch:master+topic:nova-integ+owner:Sundar | |
| 13:28:58 | sean-k-mooney | Sundar: ok so its just the nova code that is pending | |
| 13:29:07 | Sundar | There is one related patch for Nova notification https://review.opendev.org/674520 that is close | |
| 13:29:24 | aspiers | kashyap: Where did you hear that an SEV-enabled kernel would fail to boot if SEV is not provided? I have empirically proven that false by booting the same image both with and without SEV | |
| 13:29:58 | sean-k-mooney | Sundar: ok yes without that nova will timeout waiting and rollback the spawn | |
| 13:30:16 | mriedem | Sundar: no testing at all for those cyborg patches? | |
| 13:30:18 | kashyap | aspiers: DanPB; but to be fair to him, he used a qualifier "IIUC". If you've tested it, I'll go with your data | |
| 13:30:18 | Sundar | Yes. BTW, I updated the Nova patches too. Notification works with 674520. If you look at the seeming UT failures, they are mostly unrelated to Cyborg | |
| 13:30:28 | aspiers | kashyap: gotcha | |
| 13:30:44 | Sundar | mriedem: What do you mean by 'no testing'? | |
| 13:30:53 | sean-k-mooney | Sundar: have we had a end to end run of the tempest job | |
| 13:31:09 | mriedem | Sundar: there are no tests associated with that patch | |
| 13:31:12 | Sundar | The tempest CI is working with the patches, and we are hoping to merge it by this week. | |
| 13:31:17 | efried | mriedem: do we use "migration instance" for evac? | |
| 13:31:23 | mriedem | efried: no | |
| 13:31:29 | mriedem | efried: you mean migratoin-based allocations? | |
| 13:31:29 | efried | so what happens to allocations? | |
| 13:31:34 | mriedem | efried: they sit on the source | |
| 13:31:34 | efried | yeah | |
| 13:31:56 | efried | thanks | |
| 13:31:58 | mriedem | which is why we can't delete resource providers when deleting a compute service that has evacuated instance allocations against it | |
| 13:32:03 | mriedem | that whole thread i had in the ML | |
| 13:32:31 | mriedem | efried: related to this https://review.opendev.org/#/c/678100/ | |
| 13:32:41 | sean-k-mooney | mriedem: that has been changed recently right. we delete the allcoation if they exisits if and only if you bring back up the compute agent on the failed host | |
| 13:32:55 | sean-k-mooney | so that only help if you repair whatever the issue was | |
| 13:33:06 | mriedem | sean-k-mooney: yes the source allocations are deleted if you bring up the evacuated-from compute service | |
| 13:33:10 | sean-k-mooney | if you dont you get into the stiutation in that ml thread | |
| 13:33:42 | mriedem | i don't think it's probably uncommon to have a host failure, down the compute service, evacuate it, and then try to delete the comptue service before redeploying on that host | |
| 13:34:40 | mriedem | Sundar: tempest only covers happy path testing for the most part; unit tests are good for testing error conditions and such - exceptional cases | |
| 13:34:53 | mriedem | anyway, that's up to the cyborg core team for how they want to enforce testing standards | |
| 13:35:04 | efried | mriedem: also rebuild? | |
| 13:35:10 | mriedem | efried: a rebuild isn't a migration | |
| 13:35:17 | mriedem | and you can't rebuild on a down host | |
| 13:35:53 | mriedem | so the host stays the same and the flavor stays the same, but the image might change on a rebuild | |
| 13:36:00 | sean-k-mooney | Sundar: have the depencies been fixed in https://review.opendev.org/#/c/670999/ | |
| 13:36:12 | efried | mriedem: right, separate topic, I'm saying: does rebuild also end up with multiple sets of allocations for the same instance uuid? | |
| 13:36:28 | sean-k-mooney | Sundar: you rebased it but did not run the new job via "check experimental" | |
| 13:36:40 | sean-k-mooney | so we still dont have a run that worked | |
| 13:37:03 | sean-k-mooney | the previous run failded https://c3308e17743765936b80-6c7fec3fffbf24afb7394804bcdecfae.ssl.cf2.rackcdn.com/670999/7/experimental/cyborg-tempest/2fe52ec/testr_results.html.gz | |
| 13:37:12 | mriedem | efried: no | |
| 13:37:17 | sean-k-mooney | that said we knew the depencies were wrong so we expected that | |
| 13:38:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fixing broken links https://review.opendev.org/681206 | |
| 13:39:31 | luyao | mriedem: what about in the process of rebuild | |
| 13:40:08 | luyao | mriedem: rebuild is not done, how many allocations will an instance have? | |