| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-30 | |||
| 15:53:04 | sahid | i was not sure if it was something we wanted to do, so i tried to be proactive ;) | |
| 15:53:29 | melwitt | yeah, I wasn't sure either, so no worries there :) | |
| 15:53:49 | zcorneli | Works for me either way. Definitely agreed on having it all under the one bp. | |
| 16:03:36 | cfriesen | dansmith: any chance you could take a look at mriedem's patch at https://review.openstack.org/#/c/465787/ ? It's a noticeable improvement for instances that have many vifs, we've been using it for a year now. | |
| 16:04:35 | dansmith | cfriesen: I'll add it to my queue, but in the middle of something else atm | |
| 16:04:44 | cfriesen | no worries, it's waited this long. :) | |
| 16:13:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use instance project/user when creating RequestSpec during resize reschedule https://review.openstack.org/571245 | |
| 16:18:08 | melwitt | I saw that mriedem suggested another spec review day earlier and thought it sounds like a good idea. what does everyone think of a spec review day next tuesday jun 5 ahead of the spec freeze jun 7? | |
| 16:21:14 | mriedem | wfm | |
| 16:26:57 | mriedem | jaypipes: https://review.openstack.org/#/c/571245/ is part 1 in my multi-part novella about request spec + AggregateMultiTenancyIsolation + cold migrate | |
| 16:27:18 | mriedem | "it was a dark and stormy night, and we had an incomplete request spec causing much gastric distress" | |
| 16:27:21 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 16:27:27 | jaypipes | stephenfin: ^^ | |
| 16:27:54 | jaypipes | mriedem: worst. plotline. ever. | |
| 16:27:56 | jaypipes | :P | |
| 16:28:14 | mriedem | ha | |
| 16:28:19 | stephenfin | jaypipes: Sweet. It's nearly 5:30 and super sunny outside so I'm going to go roll around in the grass or something. I'll tackle that in the morning though | |
| 16:29:07 | stephenfin | jaypipes: In the interim, the NUMA aware vSwitch spec has been updated per comments from dansmith and mriedem (plus gibi). Would appreciate another look over | |
| 16:29:52 | jaypipes | stephenfin: rock on. | |
| 16:30:00 | jaypipes | stephenfin: will do. | |
| 16:33:13 | openstackgerrit | Zack Cornelius proposed openstack/nova master: Refactor libvirt get_memory_used_mb() https://review.openstack.org/571030 | |
| 16:33:14 | openstackgerrit | Zack Cornelius proposed openstack/nova master: Implement file backed memory for instances in libvirt https://review.openstack.org/567876 | |
| 16:36:36 | dansmith | mriedem: wassup with this? https://bugs.launchpad.net/nova/+bug/1675607 | |
| 16:36:38 | openstack | Launchpad bug 1675607 in OpenStack Compute (nova) "test_list_migrations_in_flavor_resize_situation fails with NoValidHost - AvailabilityZoneFilter returned 0 hosts" [High,Fix released] - Assigned to Dan Smith (danms) | |
| 16:38:34 | mriedem | dansmith: that i changed the severity | |
| 16:38:35 | mriedem | ? | |
| 16:38:41 | mriedem | see https://bugs.launchpad.net/nova/+bug/1774205 | |
| 16:38:42 | openstack | Launchpad bug 1774205 in OpenStack Compute (nova) "AggregateMultiTenancyIsolation uses wrong tenant_id during cold migrate" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 16:38:42 | dansmith | mriedem: yeah | |
| 16:38:55 | mriedem | the problem that was reported in the ops list was fixed by you via that bug in pike, | |
| 16:39:16 | mriedem | but there are still some issues like the patch above, plus i'm going to add a functional test for the AggregateMultiTenancyIsolation scenario in the ops list | |
| 16:39:16 | dansmith | oh I see | |
| 16:39:23 | dansmith | okay | |
| 16:44:15 | openstackgerrit | Chris Friesen proposed openstack/nova-specs master: Add support for emulated virtual TPM https://review.openstack.org/571111 | |
| 16:45:38 | openstackgerrit | Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597 | |
| 16:45:39 | jaypipes | efried: ^ | |
| 16:56:02 | dansmith | mriedem: are you intentionally keeping this unified for a backport or something? https://review.openstack.org/#/c/465787/7/nova/network/neutronv2/api.py@2406 | |
| 16:56:26 | dansmith | that does seem a bit like a "if special case, do thing, then bail" hack in the middle of a function | |
| 16:57:42 | mriedem | not necessarily for a backport, but to keep the scope of the change down | |
| 16:57:57 | mriedem | splitting it out into a separate function means it has to pass back a sentinel to tell the caller what to do | |
| 16:58:49 | mriedem | i can do that, i just figured it was easier in a follow up so it doesn't confuse the original fix | |
| 17:07:36 | mriedem | i can never remember if a cold migrate puts the server in VERIFY_RESIZE status | |
| 17:07:51 | mriedem | it's all the same code so i assume it does | |
| 17:08:09 | dansmith | afaik it does | |
| 17:08:20 | melwitt | last time I used it in devstack it does | |
| 17:08:31 | mriedem | our api ref doesn't mention any post-conditions on cold migrate | |
| 17:08:32 | melwitt | have to confirm or revert the resize after that | |
| 17:08:34 | mriedem | so i always have to look | |
| 17:11:08 | mriedem | low-hanging-fruit for anyone that cares https://bugs.launchpad.net/nova/+bug/1774234 | |
| 17:11:09 | openstack | Launchpad bug 1774234 in OpenStack Compute (nova) "api-ref: cold migrate reference doesn't mention asynchronous post conditions" [Medium,Triaged] | |
| 17:11:29 | melwitt | noyce | |
| 17:31:21 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 17:31:22 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add notification support for trusted_certs https://review.openstack.org/563269 | |
| 17:31:23 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 17:43:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 17:46:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 18:09:03 | mdbooth | dansmith: Hey, looking at https://bugzilla.redhat.com/show_bug.cgi?id=1584315 at first glance I think that would affect master | |
| 18:09:04 | openstack | bugzilla.redhat.com bug 1584315 in openstack-nova "DiskNotFound error during instance resize-confirm after restarting nova compute" [Unspecified,New] - Assigned to nova-maint | |
| 18:09:34 | dansmith | mdbooth: if so you should be able to find it in gate logs during a resize, right? | |
| 18:10:26 | mdbooth | dansmith: Depends how long it goes unconfirmed | |
| 18:10:44 | mdbooth | But specifically the issue after resize | |
| 18:10:58 | mdbooth | Sorry, *restart* of the service | |
| 18:11:41 | mdbooth | It looks like ResourceTracker still initially populates the compute_nodes dict in update_available_resource | |
| 18:11:50 | mdbooth | Which calls driver.get_available_resource() first | |
| 18:11:56 | mdbooth | which I think would still fail | |
| 18:12:12 | mdbooth | ... so it would never be populated | |
| 18:12:36 | mdbooth | I should.. you know... actually test it :) | |
| 18:12:47 | dansmith | if you have to restart the service while something is in resize then we wouldn't hit it in the gate, | |
| 18:12:53 | dansmith | but you could write a functional test for it | |
| 18:13:10 | mdbooth | That's part 2 of the failure. Part 1 we could potentially see in the gate | |
| 18:13:15 | mdbooth | That's the DiskNotFound error | |
| 18:13:27 | dansmith | yeah | |
| 18:13:32 | mdbooth | If an instance stayed unconfirmed for more than a minute, we should see that | |
| 18:13:52 | dansmith | or we got lucky | |
| 18:14:53 | dansmith | mdbooth: mriedem <3's helping people with logstash | |
| 18:20:59 | dansmith | mdbooth: http://logs.openstack.org/96/569296/1/check/legacy-tempest-dsvm-neutron-scenario-linuxbridge/518dd97/logs/screen-n-cpu.txt#_May_30_17_38_39_176396 | |
| 18:21:12 | mriedem1 | mdbooth: dansmith: logstash taint gonna help ye | |
| 18:21:34 | dansmith | mriedem1: no? | |
| 18:21:36 | mriedem1 | because we don't have tests in tempest that wait for periodics to run in the services | |
| 18:21:50 | mriedem1 | the tempest tests would be non-deterministic and likely timeout | |
| 18:21:52 | dansmith | mriedem1: right, luck | |
| 18:21:58 | mdbooth | Yeah | |
| 18:22:01 | dansmith | I found a bunch of cases where it happens | |
| 18:22:16 | dansmith | here's another: http://logs.openstack.org/82/566182/5/check/neutron-tempest-ovsfw/30a3077/logs/screen-n-cpu.txt#_May_30_15_33_14_686624 | |
| 18:22:16 | mdbooth | What did you type in the query box, btw? | |
| 18:22:27 | mriedem1 | i've seen things where we get DiskNotFound races while the RT update periodic runs while we're concurrently deleting a server | |
| 18:22:45 | dansmith | mdbooth: message:"DiskNotFound" | |
| 18:23:01 | mriedem1 | dansmith: yup same call stack i was thinking of | |
| 18:23:29 | dansmith | mriedem1: sounds like if you restart the service in that scenario, you get really screwed | |
| 18:23:57 | dansmith | anyway, mdbooth is going to fix it up :) | |
| 18:24:01 | mriedem1 | what we need is HA nova-compute | |
| 18:24:59 | dansmith | mriedem: mo' computes, mo' pro'lems | |
| 18:25:16 | mriedem | this is the recent thing i was thinking of https://github.com/openstack/nova/commit/5f16e714f58336344752305f94451e7c7c55742c | |
| 18:26:14 | dansmith | ah, but we shouldn't stop going to the next instance and finishing update_available_resource() | |
| 18:26:28 | mriedem | we don't after that patch | |
| 18:26:38 | mriedem | depending on the state of the instance | |
| 18:26:48 | dansmith | right, but this will happen on resize, before confirm, | |
| 18:26:51 | dansmith | which means you will re-raise | |
| 18:26:56 | dansmith | since we don't undefine the domain until confirm | |
| 18:27:48 | mriedem | sure, i was just handling the race with delete bug i was seeing in the gate all the time | |