| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-01 | |||
| 04:03:26 | mriedem | smcginnis: http://lists.openstack.org/pipermail/openstack-dev/2017-January/111295.html ? | |
| 04:03:32 | mriedem | you don't have placement-api in ENABLED_SERVICES | |
| 04:03:58 | smcginnis | mriedem: Mmmm, nope, I don't. | |
| 04:04:17 | smcginnis | mriedem: Guess I should have read that one. | |
| 04:04:28 | mriedem | dansmith: melwitt: just fyi for the morning https://review.openstack.org/#/c/427544/ | |
| 04:04:39 | mriedem | this isn't a super high failure rate so i might have some rechecking to do | |
| 04:04:43 | mriedem | but it's high enough | |
| 04:04:50 | mriedem | 164 hits since 1/27 | |
| 04:07:00 | melwitt | mriedem: thanks. I'm looking at it now. nothing jumping out at me yet | |
| 04:07:31 | mriedem | the mess of conditionals all over create/delete/get in api and conductor makes me think something can slip in here | |
| 04:07:41 | mriedem | and we're probably doing a pass or some other assumption in one of them | |
| 04:07:54 | mriedem | like, | |
| 04:07:58 | mriedem | well, | |
| 04:08:06 | mriedem | nvm | |
| 04:08:52 | mriedem | i was thinking, before the api casts to conductor to create the instance in the cell, the build request could be deleted, and then we create the instance from stale build request info, but then in conductor when we go to delete the build request, it should fail with a 404 and then we should cleanup the instance too | |
| 04:09:24 | mriedem | however, i wonder if the GET request is hitting right in between the time that the instance is created and the time that we try to delete the build request and find out it's already deleted, then cleanup | |
| 04:09:32 | melwitt | hmm | |
| 04:09:38 | mriedem | because there is no mutex there at all from what i can tell, | |
| 04:09:55 | mriedem | and after we cast to conductor, we never refresh the build request from the db to get it's current state before we actually create the instance in the cell, from what i can tell anyway | |
| 04:12:56 | melwitt | yeah, that sounds reasonable. I was thinking along those lines of somehow the instance getting resurrected | |
| 04:17:02 | openstackgerrit | Matt Riedemann proposed openstack/nova: WIP: Add debug logging for delete/query race bug https://review.openstack.org/427544 | |
| 04:21:10 | mriedem | well i'm going to call it a night i guess | |
| 04:21:14 | mriedem | maybe we'll catch a fish | |
| 04:21:32 | melwitt | gnight o/ | |
| 05:08:57 | openstackgerrit | Yoshihiko Atsumi proposed openstack/nova: Add resize server functional negative tests https://review.openstack.org/420514 | |
| 07:22:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova: Refactor a test method including 7 test cases https://review.openstack.org/395436 | |
| 07:22:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova: api-ref: Fix keypair API parameters https://review.openstack.org/420626 | |
| 07:46:55 | lennyb | wznoinsk, I see a lot of multinode jobs fail lately (also our), I found that it's related to latest placement-api that was merged. | |
| 08:00:17 | lennyb | Infra, regarding gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial where can I find nova.conf files for hypervisors of this job. I have some troubles to apply placement api to our multinode setup | |
| 09:26:24 | wznoinsk | lennyb, our people are already on it :-), thanks for pinging | |
| 09:27:33 | lyarwood | johnthetubaguy, sfinucan, bauzas ; Would you mind taking a look at https://review.openstack.org/#/c/427267/ this morning, mriedem marked the bug as High / ocata-rc-potential so I'd like to get a few more reviews on it today if possible. | |
| 09:35:56 | oomichi | mriedem: yeah, 422171 becomes better | |
| 09:37:03 | bauzas | lyarwood: roger | |
| 09:42:27 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova: Optimize time during PCI alias parsing process https://review.openstack.org/427145 | |
| 10:01:23 | lennyb | wznoinsk, looks like your people should add placement-api to enabled services on all computes | |
| 10:01:59 | wznoinsk | lennyb, maybe they did already, I'll pass the hint, thanks | |
| 10:02:19 | gmann | johnthetubaguy: alex_xu : can you check(whenever get time), i think we cannot change default policy value directly - https://review.openstack.org/#/c/396420/5 | |
| 10:03:06 | gmann | johnthetubaguy: alex_xu also if you can check this sort/filter release note. I updated this one - https://review.openstack.org/#/c/421760/ | |
| 10:08:44 | johnthetubaguy | gmann: I think I am +1 a lot of your comments, I am going to draft a spec around the policy changes I would like to see happen next cycle | |
| 10:09:51 | gmann | johnthetubaguy: yea that nice. | |
| 10:10:13 | gmann | johnthetubaguy: i tried to update the existing one but yea separate is better-https://review.openstack.org/#/c/425533/ | |
| 10:10:14 | johnthetubaguy | working with some folks over in keystone land on that, with any luck, they have similar ideas/plans forming | |
| 10:10:31 | gmann | ok | |
| 10:10:54 | johnthetubaguy | want to get stuff laid out with some structure so we can talk through things at the PTG, with any luck. | |
| 10:11:43 | johnthetubaguy | lyarwood: ack | |
| 10:12:02 | gmann | yea that is more aligned for PTG and prepare before that | |
| 10:12:47 | gmann | johnthetubaguy: alex_xu btw do we have meeting today? do not think alex_xu is back | |
| 10:13:55 | johnthetubaguy | gmann: I don't know if we decided something, given its crazy o'clock for your all, I vote we skip it, but I will try start the meeting in case folks are about and want to discuss things | |
| 10:15:33 | gmann | ok | |
| 10:20:12 | openstack | Launchpad bug 1605016 in OpenStack Compute (nova) "Post copy live migration interrupts network connectivity" [High,In progress] - Assigned to Sivasathurappan Radhakrishnan (siva-radhakrishnan) | |
| 10:20:12 | kashyap | raj_singh: Hi, you have assigned someone to work on this -- https://bugs.launchpad.net/nova/+bug/1605016 | |
| 10:20:18 | cdent | gmann, johnthetubaguy: If we're talking about api meetings I'd vote for skip too | |
| 10:20:31 | kashyap | Do they have requisite knowledge to work on it? Or should I reassign the bug back to 'nodbody'? | |
| 10:22:42 | gmann | cdent: yea, api meeting. me too to skip. | |
| 10:27:20 | johnthetubaguy | kashyap: we are training him up, he is struggling a bit on it though | |
| 10:27:53 | kashyap | johnthetubaguy: Me nods. That bug is quite complicated. Maybe a lower-hanging fruit might be better? | |
| 10:27:57 | johnthetubaguy | kashyap: he is testing out the fix in an environment at the moment I believe | |
| 10:28:05 | kashyap | johnthetubaguy: I say that because, right now QEMU migration devs are asking: What is Nova's stance on postcopy | |
| 10:28:13 | kashyap | So, I lost complete memory of it | |
| 10:28:19 | kashyap | Maybe mdbooth can shed some light there? | |
| 10:28:33 | johnthetubaguy | kashyap: support is in, works with majority of neutron back ends, but not all, AFAIK | |
| 10:29:06 | johnthetubaguy | we discussed it alot at the summit, and siva_krishnan is implementing what we agreed would be the "quick fix", calling neutron earlier in the process | |
| 10:29:41 | mdbooth | johnthetubaguy: IIRC the problem with that change wasn't so much working out what needed to be done, as finding somebody with enough arcane neutron knowledge to know which backends allowed it to be done. | |
| 10:29:49 | johnthetubaguy | would love to trigger of QEMU events to get more exact timing of when the post copy occurs, but the basic structure is going in the above change | |
| 10:30:23 | johnthetubaguy | mdbooth: I am confused then, neutron folks agreed this should be OK at the summit, what changed from that? | |
| 10:30:32 | mdbooth | johnthetubaguy: The qemu events things isn't required at all in this instance, because the change happens synchronously on the source. | |
| 10:30:41 | kashyap | johnthetubaguy: I have asked David Gilbert of QEMU to join here. He can ask questions to his heart's content, and he goes out of his way to explain | |
| 10:30:44 | davidgiluk | kashyap: nod | |
| 10:30:54 | kashyap | davidgiluk: Ah, you are here :-) | |
| 10:30:55 | mdbooth | i.e. there's no need to respond to an event from a call which you just made in code you're still executing | |
| 10:31:00 | johnthetubaguy | mdbooth: I keep being told its not syncronus | |
| 10:31:13 | mdbooth | What's not synchronous about it? | |
| 10:31:19 | johnthetubaguy | you just say, please do post copy at end of next iterration, I believe | |
| 10:31:30 | johnthetubaguy | so when we call post copy, it doesn't immediatly happen, I was told | |
| 10:31:31 | kashyap | davidgiluk: So, this is what johnthetubaguy was saying, when I asked him about "What is Nova's stance on postcopy": < johnthetubaguy> support is in, works with majority of neutron back ends, but not all, AFAIK | |
| 10:32:13 | johnthetubaguy | there can be a small network outage with some backends, during the post copy process (which ironically speeds up the live-migrate, of course) doing work to avoid that | |
| 10:32:24 | mdbooth | johnthetubaguy: In _live_migration_monitor we do: | |
| 10:32:24 | openstack | Launchpad bug 1605016 in OpenStack Compute (nova) "Post copy live migration interrupts network connectivity" [High,In progress] https://launchpad.net/bugs/1605016 - Assigned to Sivasathurappan Radhakrishnan (siva-radhakrishnan) | |
| 10:32:24 | davidgiluk | kashyap: Yeh so my understanding was it was waiting on a fix to lp 1605016 for the others, with https://review.openstack.org/#/c/413021/9 being a proposed fix | |
| 10:32:32 | mdbooth | libvirt_migrate.trigger_postcopy_switch(guest, | |
| 10:32:32 | mdbooth | previous_data_remaining, migration.status)): | |
| 10:32:32 | mdbooth | info.memory_iteration, info.data_remaining, | |
| 10:32:32 | mdbooth | libvirt_migrate.should_switch_to_postcopy( | |
| 10:32:32 | mdbooth | if (is_post_copy_enabled and | |
| 10:32:33 | mdbooth | instance, | |
| 10:32:35 | mdbooth | migration) | |
| 10:32:51 | johnthetubaguy | mdbooth: its the libvirt call that I am told is async | |
| 10:33:07 | johnthetubaguy | mdbooth: it just request post copy at some point in the future, and returns right away, before actually triggering it | |
| 10:33:12 | mdbooth | johnthetubaguy: Well we might switch a few milliseconds early | |
| 10:33:16 | johnthetubaguy | mdbooth: pkoniszewski has the details on that | |
| 10:33:19 | mdbooth | Rather than a few milliseconds late | |
| 10:33:39 | johnthetubaguy | mdbooth: I was told more like seconds, but yeah, I was betting on milliseconds myself | |
| 10:33:46 | davidgiluk | the postcopy transition happens at the end of a pass, and libvirt gets involved before starting the destination | |
| 10:34:03 | johnthetubaguy | mdbooth: either way, I vote for the approach siva_krishnan has in the works already, just need to iron things out | |
| 10:34:12 | johnthetubaguy | davidgiluk: sorry, whats waiting on the fix/ | |
| 10:34:23 | johnthetubaguy | s/ / / ? / | |
| 10:34:32 | davidgiluk | johnthetubaguy: The support for other networking configs | |
| 10:35:05 | johnthetubaguy | davidgiluk: that bug fix should help reduce the network outage some networking configs currently see, yes | |
| 10:35:43 | davidgiluk | johnthetubaguy: < johnthetubaguy> would love to trigger of QEMU events to get more exact timing of when the post copy occurs, but the basic structure is going in the above change | |
| 10:35:48 | davidgiluk | johnthetubaguy: We already emit the events | |