| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-30 | |||
| 16:39:16 | mriedem | but there are still some issues like the patch above, plus i'm going to add a functional test for the AggregateMultiTenancyIsolation scenario in the ops list | |
| 16:39:16 | dansmith | oh I see | |
| 16:39:23 | dansmith | okay | |
| 16:44:15 | openstackgerrit | Chris Friesen proposed openstack/nova-specs master: Add support for emulated virtual TPM https://review.openstack.org/571111 | |
| 16:45:38 | openstackgerrit | Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597 | |
| 16:45:39 | jaypipes | efried: ^ | |
| 16:56:02 | dansmith | mriedem: are you intentionally keeping this unified for a backport or something? https://review.openstack.org/#/c/465787/7/nova/network/neutronv2/api.py@2406 | |
| 16:56:26 | dansmith | that does seem a bit like a "if special case, do thing, then bail" hack in the middle of a function | |
| 16:57:42 | mriedem | not necessarily for a backport, but to keep the scope of the change down | |
| 16:57:57 | mriedem | splitting it out into a separate function means it has to pass back a sentinel to tell the caller what to do | |
| 16:58:49 | mriedem | i can do that, i just figured it was easier in a follow up so it doesn't confuse the original fix | |
| 17:07:36 | mriedem | i can never remember if a cold migrate puts the server in VERIFY_RESIZE status | |
| 17:07:51 | mriedem | it's all the same code so i assume it does | |
| 17:08:09 | dansmith | afaik it does | |
| 17:08:20 | melwitt | last time I used it in devstack it does | |
| 17:08:31 | mriedem | our api ref doesn't mention any post-conditions on cold migrate | |
| 17:08:32 | melwitt | have to confirm or revert the resize after that | |
| 17:08:34 | mriedem | so i always have to look | |
| 17:11:08 | mriedem | low-hanging-fruit for anyone that cares https://bugs.launchpad.net/nova/+bug/1774234 | |
| 17:11:09 | openstack | Launchpad bug 1774234 in OpenStack Compute (nova) "api-ref: cold migrate reference doesn't mention asynchronous post conditions" [Medium,Triaged] | |
| 17:11:29 | melwitt | noyce | |
| 17:31:21 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 17:31:22 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add notification support for trusted_certs https://review.openstack.org/563269 | |
| 17:31:23 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 17:43:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 17:46:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 18:09:03 | mdbooth | dansmith: Hey, looking at https://bugzilla.redhat.com/show_bug.cgi?id=1584315 at first glance I think that would affect master | |
| 18:09:04 | openstack | bugzilla.redhat.com bug 1584315 in openstack-nova "DiskNotFound error during instance resize-confirm after restarting nova compute" [Unspecified,New] - Assigned to nova-maint | |
| 18:09:34 | dansmith | mdbooth: if so you should be able to find it in gate logs during a resize, right? | |
| 18:10:26 | mdbooth | dansmith: Depends how long it goes unconfirmed | |
| 18:10:44 | mdbooth | But specifically the issue after resize | |
| 18:10:58 | mdbooth | Sorry, *restart* of the service | |
| 18:11:41 | mdbooth | It looks like ResourceTracker still initially populates the compute_nodes dict in update_available_resource | |
| 18:11:50 | mdbooth | Which calls driver.get_available_resource() first | |
| 18:11:56 | mdbooth | which I think would still fail | |
| 18:12:12 | mdbooth | ... so it would never be populated | |
| 18:12:36 | mdbooth | I should.. you know... actually test it :) | |
| 18:12:47 | dansmith | if you have to restart the service while something is in resize then we wouldn't hit it in the gate, | |
| 18:12:53 | dansmith | but you could write a functional test for it | |
| 18:13:10 | mdbooth | That's part 2 of the failure. Part 1 we could potentially see in the gate | |
| 18:13:15 | mdbooth | That's the DiskNotFound error | |
| 18:13:27 | dansmith | yeah | |
| 18:13:32 | mdbooth | If an instance stayed unconfirmed for more than a minute, we should see that | |
| 18:13:52 | dansmith | or we got lucky | |
| 18:14:53 | dansmith | mdbooth: mriedem <3's helping people with logstash | |
| 18:20:59 | dansmith | mdbooth: http://logs.openstack.org/96/569296/1/check/legacy-tempest-dsvm-neutron-scenario-linuxbridge/518dd97/logs/screen-n-cpu.txt#_May_30_17_38_39_176396 | |
| 18:21:12 | mriedem1 | mdbooth: dansmith: logstash taint gonna help ye | |
| 18:21:34 | dansmith | mriedem1: no? | |
| 18:21:36 | mriedem1 | because we don't have tests in tempest that wait for periodics to run in the services | |
| 18:21:50 | mriedem1 | the tempest tests would be non-deterministic and likely timeout | |
| 18:21:52 | dansmith | mriedem1: right, luck | |
| 18:21:58 | mdbooth | Yeah | |
| 18:22:01 | dansmith | I found a bunch of cases where it happens | |
| 18:22:16 | dansmith | here's another: http://logs.openstack.org/82/566182/5/check/neutron-tempest-ovsfw/30a3077/logs/screen-n-cpu.txt#_May_30_15_33_14_686624 | |
| 18:22:16 | mdbooth | What did you type in the query box, btw? | |
| 18:22:27 | mriedem1 | i've seen things where we get DiskNotFound races while the RT update periodic runs while we're concurrently deleting a server | |
| 18:22:45 | dansmith | mdbooth: message:"DiskNotFound" | |
| 18:23:01 | mriedem1 | dansmith: yup same call stack i was thinking of | |
| 18:23:29 | dansmith | mriedem1: sounds like if you restart the service in that scenario, you get really screwed | |
| 18:23:57 | dansmith | anyway, mdbooth is going to fix it up :) | |
| 18:24:01 | mriedem1 | what we need is HA nova-compute | |
| 18:24:59 | dansmith | mriedem: mo' computes, mo' pro'lems | |
| 18:25:16 | mriedem | this is the recent thing i was thinking of https://github.com/openstack/nova/commit/5f16e714f58336344752305f94451e7c7c55742c | |
| 18:26:14 | dansmith | ah, but we shouldn't stop going to the next instance and finishing update_available_resource() | |
| 18:26:28 | mriedem | we don't after that patch | |
| 18:26:38 | mriedem | depending on the state of the instance | |
| 18:26:48 | dansmith | right, but this will happen on resize, before confirm, | |
| 18:26:51 | dansmith | which means you will re-raise | |
| 18:26:56 | dansmith | since we don't undefine the domain until confirm | |
| 18:27:48 | mriedem | sure, i was just handling the race with delete bug i was seeing in the gate all the time | |
| 18:27:55 | dansmith | yup | |
| 18:27:59 | mriedem | i figured there would be additional scenarios that need to be dealt with | |
| 18:28:05 | mriedem | once mdbooth smelt what was dealt | |
| 18:30:25 | mdbooth | I don't think that patch will fix this, because it only ignores DiskNotFound if task state is set | |
| 18:30:39 | mdbooth | Which it looks like it's not between resize and confirm | |
| 18:30:41 | dansmith | mdbooth: rightI just said that :) | |
| 18:30:53 | mdbooth | dansmith: Well I wasn't looking! I was reading code. | |
| 18:30:55 | dansmith | mdbooth: that patch was for the deleting task state basically | |
| 18:30:59 | dansmith | mdbooth: keep up! | |
| 18:50:59 | melwitt | dansmith, mriedem: are we cells meeting today? looks like the CERN peeps aren't around today | |
| 18:51:17 | dansmith | oh, I have something on my calendar that covers it up so I wasn't paying attention | |
| 18:51:23 | dansmith | so yeah, cancel if we can | |
| 18:51:26 | dansmith | we just talked last week anyway | |
| 18:51:48 | melwitt | +1 | |
| 18:53:28 | mriedem | don't need to have a meeting, but tssurya has 2 bug fixes that i'm +2 on which you should probably look at | |
| 18:53:38 | melwitt | k | |
| 18:53:41 | dansmith | links | |
| 18:53:52 | mriedem | https://review.openstack.org/#/c/568271/ and https://review.openstack.org/#/c/567878/ | |
| 18:54:32 | dansmith | I'll get the second | |
| 18:54:38 | dansmith | since online migration | |
| 18:54:56 | melwitt | k | |
| 19:05:41 | mriedem | dansmith: we have backported online data migrations to fix bugs before https://review.openstack.org/#/c/408725/ | |
| 19:06:41 | dansmith | okay, probably want to do a special reno with it though right? so people (if paying attention) will know to re-run them | |
| 19:08:26 | mriedem | yeah i think so, | |
| 19:08:35 | mriedem | https://review.openstack.org/#/c/529389/ was similar but not as explicit, | |
| 19:09:01 | mriedem | the idea just being, if we can get the fix into the stable branches before people roll up to them, then great, else we tried | |
| 19:09:17 | dansmith | sure, that's a fair point | |
| 19:09:44 | dansmith | anyway, wasn't saying we couldn't or shouldn't, just that we should highlight it | |
| 19:09:45 | mriedem | should we put a simple 'fixes' reno on that patch now before backports? | |
| 19:09:57 | mriedem | then we don't need to make it weird on stable | |