| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-30 | |||
| 17:11:09 | openstack | Launchpad bug 1774234 in OpenStack Compute (nova) "api-ref: cold migrate reference doesn't mention asynchronous post conditions" [Medium,Triaged] | |
| 17:11:29 | melwitt | noyce | |
| 17:31:21 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 17:31:22 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add notification support for trusted_certs https://review.openstack.org/563269 | |
| 17:31:23 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 17:43:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 17:46:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 18:09:03 | mdbooth | dansmith: Hey, looking at https://bugzilla.redhat.com/show_bug.cgi?id=1584315 at first glance I think that would affect master | |
| 18:09:04 | openstack | bugzilla.redhat.com bug 1584315 in openstack-nova "DiskNotFound error during instance resize-confirm after restarting nova compute" [Unspecified,New] - Assigned to nova-maint | |
| 18:09:34 | dansmith | mdbooth: if so you should be able to find it in gate logs during a resize, right? | |
| 18:10:26 | mdbooth | dansmith: Depends how long it goes unconfirmed | |
| 18:10:44 | mdbooth | But specifically the issue after resize | |
| 18:10:58 | mdbooth | Sorry, *restart* of the service | |
| 18:11:41 | mdbooth | It looks like ResourceTracker still initially populates the compute_nodes dict in update_available_resource | |
| 18:11:50 | mdbooth | Which calls driver.get_available_resource() first | |
| 18:11:56 | mdbooth | which I think would still fail | |
| 18:12:12 | mdbooth | ... so it would never be populated | |
| 18:12:36 | mdbooth | I should.. you know... actually test it :) | |
| 18:12:47 | dansmith | if you have to restart the service while something is in resize then we wouldn't hit it in the gate, | |
| 18:12:53 | dansmith | but you could write a functional test for it | |
| 18:13:10 | mdbooth | That's part 2 of the failure. Part 1 we could potentially see in the gate | |
| 18:13:15 | mdbooth | That's the DiskNotFound error | |
| 18:13:27 | dansmith | yeah | |
| 18:13:32 | mdbooth | If an instance stayed unconfirmed for more than a minute, we should see that | |
| 18:13:52 | dansmith | or we got lucky | |
| 18:14:53 | dansmith | mdbooth: mriedem <3's helping people with logstash | |
| 18:20:59 | dansmith | mdbooth: http://logs.openstack.org/96/569296/1/check/legacy-tempest-dsvm-neutron-scenario-linuxbridge/518dd97/logs/screen-n-cpu.txt#_May_30_17_38_39_176396 | |
| 18:21:12 | mriedem1 | mdbooth: dansmith: logstash taint gonna help ye | |
| 18:21:34 | dansmith | mriedem1: no? | |
| 18:21:36 | mriedem1 | because we don't have tests in tempest that wait for periodics to run in the services | |
| 18:21:50 | mriedem1 | the tempest tests would be non-deterministic and likely timeout | |
| 18:21:52 | dansmith | mriedem1: right, luck | |
| 18:21:58 | mdbooth | Yeah | |
| 18:22:01 | dansmith | I found a bunch of cases where it happens | |
| 18:22:16 | mdbooth | What did you type in the query box, btw? | |
| 18:22:16 | dansmith | here's another: http://logs.openstack.org/82/566182/5/check/neutron-tempest-ovsfw/30a3077/logs/screen-n-cpu.txt#_May_30_15_33_14_686624 | |
| 18:22:27 | mriedem1 | i've seen things where we get DiskNotFound races while the RT update periodic runs while we're concurrently deleting a server | |
| 18:22:45 | dansmith | mdbooth: message:"DiskNotFound" | |
| 18:23:01 | mriedem1 | dansmith: yup same call stack i was thinking of | |
| 18:23:29 | dansmith | mriedem1: sounds like if you restart the service in that scenario, you get really screwed | |
| 18:23:57 | dansmith | anyway, mdbooth is going to fix it up :) | |
| 18:24:01 | mriedem1 | what we need is HA nova-compute | |
| 18:24:59 | dansmith | mriedem: mo' computes, mo' pro'lems | |
| 18:25:16 | mriedem | this is the recent thing i was thinking of https://github.com/openstack/nova/commit/5f16e714f58336344752305f94451e7c7c55742c | |
| 18:26:14 | dansmith | ah, but we shouldn't stop going to the next instance and finishing update_available_resource() | |
| 18:26:28 | mriedem | we don't after that patch | |
| 18:26:38 | mriedem | depending on the state of the instance | |
| 18:26:48 | dansmith | right, but this will happen on resize, before confirm, | |
| 18:26:51 | dansmith | which means you will re-raise | |
| 18:26:56 | dansmith | since we don't undefine the domain until confirm | |
| 18:27:48 | mriedem | sure, i was just handling the race with delete bug i was seeing in the gate all the time | |
| 18:27:55 | dansmith | yup | |
| 18:27:59 | mriedem | i figured there would be additional scenarios that need to be dealt with | |
| 18:28:05 | mriedem | once mdbooth smelt what was dealt | |
| 18:30:25 | mdbooth | I don't think that patch will fix this, because it only ignores DiskNotFound if task state is set | |
| 18:30:39 | mdbooth | Which it looks like it's not between resize and confirm | |
| 18:30:41 | dansmith | mdbooth: rightI just said that :) | |
| 18:30:53 | mdbooth | dansmith: Well I wasn't looking! I was reading code. | |
| 18:30:55 | dansmith | mdbooth: that patch was for the deleting task state basically | |
| 18:30:59 | dansmith | mdbooth: keep up! | |
| 18:50:59 | melwitt | dansmith, mriedem: are we cells meeting today? looks like the CERN peeps aren't around today | |
| 18:51:17 | dansmith | oh, I have something on my calendar that covers it up so I wasn't paying attention | |
| 18:51:23 | dansmith | so yeah, cancel if we can | |
| 18:51:26 | dansmith | we just talked last week anyway | |
| 18:51:48 | melwitt | +1 | |
| 18:53:28 | mriedem | don't need to have a meeting, but tssurya has 2 bug fixes that i'm +2 on which you should probably look at | |
| 18:53:38 | melwitt | k | |
| 18:53:41 | dansmith | links | |
| 18:53:52 | mriedem | https://review.openstack.org/#/c/568271/ and https://review.openstack.org/#/c/567878/ | |
| 18:54:32 | dansmith | I'll get the second | |
| 18:54:38 | dansmith | since online migration | |
| 18:54:56 | melwitt | k | |
| 19:05:41 | mriedem | dansmith: we have backported online data migrations to fix bugs before https://review.openstack.org/#/c/408725/ | |
| 19:06:41 | dansmith | okay, probably want to do a special reno with it though right? so people (if paying attention) will know to re-run them | |
| 19:08:26 | mriedem | yeah i think so, | |
| 19:08:35 | mriedem | https://review.openstack.org/#/c/529389/ was similar but not as explicit, | |
| 19:09:01 | mriedem | the idea just being, if we can get the fix into the stable branches before people roll up to them, then great, else we tried | |
| 19:09:17 | dansmith | sure, that's a fair point | |
| 19:09:44 | dansmith | anyway, wasn't saying we couldn't or shouldn't, just that we should highlight it | |
| 19:09:45 | mriedem | should we put a simple 'fixes' reno on that patch now before backports? | |
| 19:09:57 | mriedem | then we don't need to make it weird on stable | |
| 19:10:26 | dansmith | I think a dedicated reno for the backports will make it more clear, right? "If you are applying this update after already being on the current release, you'll need to re-run online migrations" or something | |
| 19:11:21 | mriedem | sure that works for me as well | |
| 19:11:24 | dansmith | if people see that it's fixed in the stable they're running, but it's still broken for them and they don't realize they would need to re-run online migrations at a peri-release boundary they might be confused | |
| 19:12:20 | mriedem | i'll push the backport to queens once i get done with reviewing sahid's trusted vf series again | |
| 19:19:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: configure trust mode for vfs https://review.openstack.org/458514 | |
| 19:19:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: add vf_trusted field for network metadata https://review.openstack.org/566343 | |
| 19:19:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: metadata: add vf_trusted field to device metadata https://review.openstack.org/566344 | |
| 19:20:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: network: update pci request spec to handle trusted tags https://review.openstack.org/458820 | |
| 19:44:29 | efried | jaypipes: Done. mriedem: I'm +2 on https://review.openstack.org/#/c/553597/ now. | |
| 19:46:40 | mriedem | artom: you might want to take a quick run through this https://review.openstack.org/#/c/566344/ | |
| 19:46:47 | mriedem | efried: ok | |
| 19:47:38 | mriedem | "Snarklepuss is actually my middle name, not my surname." blows my mind | |
| 19:47:52 | mriedem | that's professional snark at a PhD level | |
| 19:48:25 | artom | mriedem, indeed, thanks for motivating me | |
| 19:48:38 | artom | That series has been sitting on my review list since forever | |
| 19:49:15 | mriedem | artom: besides that one patch i'm +2 on the rest | |
| 19:49:30 | mriedem | artom: if you're so inclined, you could even address my comments in that one and we could get the entire thing +2ed today | |
| 19:49:50 | artom | mriedem, not a bad idea, sahid's done for the day anyways | |
| 19:50:04 | artom | I don't think he'll mind | |