| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-20 | |||
| 13:59:52 | mriedem | jaypipes: yes i can dig up the link | |
| 14:00:01 | jaypipes | kashyap: I think you can probably afford more than just a piece of bread for lunch. | |
| 14:00:02 | kashyap | Should be back soon | |
| 14:00:06 | mriedem | jaypipes: https://review.openstack.org/#/c/465160/ | |
| 14:00:17 | jaypipes | kashyap: clearly, Holland has become too expensive. | |
| 14:00:19 | kashyap | jaypipes: Haha, I have the spreads at home; it's the Italian Ciabatta bread :P | |
| 14:00:26 | jaypipes | :) | |
| 14:00:31 | mriedem | "the spreads" | |
| 14:00:34 | mriedem | sounds like "the runs" | |
| 14:00:35 | jaypipes | ah, mriedem that's the ticket. thanks! | |
| 14:00:36 | kashyap | jaypipes: Gentleman, it's called Gent -- http://wikitravel.org/en/Gent | |
| 14:00:43 | kashyap | s/men/man/ | |
| 14:00:47 | jaypipes | heh | |
| 14:00:56 | kashyap | (It's in Flanders, Belgium :P) | |
| 14:01:58 | kashyap | It's late lunch; was debugging the Gate migration bug from hell | |
| 14:02:02 | kashyap | While I'm here; I'll update lyarwood too | |
| 14:02:10 | kashyap | lyarwood: On that migration bug we were talking about | |
| 14:02:19 | kashyap | I spent past 2 hours creating a reproducer -- | |
| 14:02:49 | kashyap | Migrate a guest with a `dd if=/dev/zero bs=1M | md5sum` workload in it, AND throttle the bandwitdh to 1MiB/s | |
| 14:02:59 | kashyap | And also set the clock back 20 minutes on the source | |
| 14:02:59 | stephenfin | jaypipes: Why do you ask? I was waiting on sean-k-mooney to wade in on that again before doing anything, personally | |
| 14:03:07 | kashyap | To see if those keepalives will kick in | |
| 14:03:19 | kashyap | s/keepalives/keep-alive timeout/ | |
| 14:03:54 | jaypipes | stephenfin: mostly I was just looking for a link to that other patch :) I couldn't remember what the commit summary was and couldn't find it on gerrit. | |
| 14:03:56 | kashyap | I couldn't reproduce it so far; the guest migrated slowly after a long while. So, the investigation to be conitnued (assuming it's still a problem for the stable gate) | |
| 14:11:46 | danpawlik | mriedem: thanks for review the patch for mkisofs calls | |
| 14:11:59 | mdbooth | Incoming | |
| 14:12:10 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | |
| 14:12:11 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | |
| 14:12:11 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | |
| 14:12:12 | openstackgerrit | Matthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests https://review.openstack.org/527916 | |
| 14:12:12 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | |
| 14:12:13 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass disk_info dict to libvirt_info https://review.openstack.org/529329 | |
| 14:12:13 | openstackgerrit | Matthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info https://review.openstack.org/529328 | |
| 14:13:08 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: Provide an RFB security proxy implementation https://review.openstack.org/345399 | |
| 14:13:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Document TLS security setup for noVNC proxy https://review.openstack.org/500544 | |
| 14:16:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Workaround missing RequestSpec.project_id when moving an instance https://review.openstack.org/529185 | |
| 14:16:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use instance.project_id when creating request specs for old instances https://review.openstack.org/529184 | |
| 14:16:49 | mriedem | jaypipes: i had to fix the test for py35 ^ otherwise unchanged | |
| 14:17:14 | mriedem | stephenfin: gibi: ^ contains a fix for a regression introduced in newton which breaks us later in pike, | |
| 14:17:21 | mriedem | so i'm hoping to get that backported | |
| 14:17:36 | mriedem | before we eol newton | |
| 14:17:43 | stephenfin | Looking | |
| 14:18:04 | mriedem | the workaround patch will go back to pike, | |
| 14:18:08 | mriedem | the other one can go to newton | |
| 14:20:18 | ameeda | Hello, Can you please review my code here ? https://review.openstack.org/#/c/528069/ | |
| 14:23:47 | stephenfin | ameeda: If you could add a little more context to that, it would make it much easier to review :) | |
| 14:29:11 | stephenfin | mriedem: Two questions left on https://review.openstack.org/#/c/529185/. I can ignore them if you really want to get it in now (or I've misunderstood something) | |
| 14:29:18 | stephenfin | The first one is a-ok | |
| 14:31:41 | ameeda | stephenfin: thanks for comment, I change the commit and I will submit it now | |
| 14:32:25 | mriedem | stephenfin: replied | |
| 14:34:41 | stephenfin | Cool. +2 to that too, in that case | |
| 14:39:45 | mriedem | stephenfin: thanks | |
| 14:39:51 | openstack | Launchpad bug 1739323 in OpenStack Compute (nova) pike "KeyError in host_manager for _get_host_states" [High,Confirmed] | |
| 14:39:51 | mriedem | mnaser: i'm at a loss for how you're hittin this https://bugs.launchpad.net/nova/+bug/1739323 | |
| 14:40:19 | mriedem | mnaser: dug through it again and the entries in seen_nodes should all also be in host_state_map because of https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/scheduler/host_manager.py#L698 | |
| 14:40:50 | mriedem | and i don't see anything else in the HostStateManager that modifies that dict, so no locking issues that i can see, unless you have a multiple workers patch or something | |
| 14:41:57 | mriedem | mdbooth: maybe your keen eye can spot an obvious issue in here ^ | |
| 14:50:04 | openstackgerrit | Ameed Ashour proposed openstack/nova master: detach instance volumes when VM creation fails https://review.openstack.org/528385 | |
| 14:50:04 | openstackgerrit | Ameed Ashour proposed openstack/nova master: detaching volumes when deleted VM is in error state https://review.openstack.org/528069 | |
| 14:55:41 | ameeda | stephenfin: please check the comment and let me know, Thanks | |
| 14:57:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanups for the scheduler code https://review.openstack.org/529343 | |
| 15:00:52 | mdbooth | mriedem_away: I'm a little lacking in scheduler context. It seems to me that we would hit this if the scheduler was capable of evaluating filters 'concurrently'. | |
| 15:01:11 | mdbooth | For some pythonic/eventlet definition of concurrently, obviously. | |
| 15:01:36 | stephenfin | lyarwood, mdbooth: Think you could take a look at this at some point? BDM'y things https://review.openstack.org/#/c/528069 | |
| 15:01:39 | mdbooth | Does the scheduler serialize all scheduling requests? | |
| 15:04:51 | jaypipes | mdbooth: no | |
| 15:05:34 | mdbooth | jaypipes: Thanks. That looks like a bug, then. | |
| 15:05:53 | jaypipes | mdbooth: what's that? | |
| 15:06:14 | mdbooth | https://bugs.launchpad.net/nova/+bug/1739323/ | |
| 15:06:15 | openstack | Launchpad bug 1739323 in OpenStack Compute (nova) "KeyError in host_manager for _get_host_states" [High,Incomplete] | |
| 15:06:21 | jaypipes | ah | |
| 15:06:25 | jaypipes | lemme looksie | |
| 15:06:34 | mdbooth | _get_host_states is returning an iterator over global state | |
| 15:06:52 | mdbooth | So if any 2 'threads' can be calling it simultaneously, it's potentially corrupt | |
| 15:07:01 | openstackgerrit | wes hayutin proposed openstack/nova master: DNM, testing only https://review.openstack.org/529349 | |
| 15:07:18 | mdbooth | Especially as it's returning an iterator, so the evaluation period is extended | |
| 15:08:12 | mdbooth | I guess you'd expect to see this if you delete a compute node on a busy system? | |
| 15:08:17 | mdbooth | mnaser: ^^^ ? | |
| 15:09:05 | jaypipes | mdbooth: _get_host_states is returning a tuple, not an iterator. | |
| 15:09:22 | mdbooth | jaypipes: This line: | |
| 15:09:34 | mdbooth | return (self.host_state_map[host] for host in seen_nodes) | |
| 15:09:41 | jaypipes | is a tuple. | |
| 15:09:42 | mdbooth | That's a generator expression, no? | |
| 15:09:59 | jaypipes | I don't believe so... | |
| 15:10:14 | edleafe | jaypipes: it's a generator expression | |
| 15:10:17 | jaypipes | a generator would yield or return a function that yield'd, no? | |
| 15:10:58 | mdbooth | The expression itself is a generator | |
| 15:11:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add PCI NUMA policy fields https://review.openstack.org/527470 | |
| 15:11:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Modify signature of _filter_non_requested_pfs https://review.openstack.org/527473 | |
| 15:11:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add PCI NUMA policies https://review.openstack.org/527472 | |
| 15:11:17 | edleafe | jaypipes: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L139-L143 | |
| 15:11:47 | edleafe | it calls it an iterator, which is technically correct, but it's a generator | |
| 15:11:55 | mdbooth | jaypipes: So [foo for foo in foos] is a list comprehension | |
| 15:11:56 | stephenfin | cfriesen, bauzas: Fancy taking a look at ^^^ again? Has been changed quite a lot (hopefully for the better) | |
| 15:12:11 | mdbooth | And (foo for foo in foos) is a generator expression which iterates over the same thing | |
| 15:13:37 | jaypipes | hmm, interesting. | |
| 15:14:34 | mdbooth | So if that last line is returning a generator, it's essentially a closure over seen_nodes, which is local, and self.host_state_map, which is global. | |
| 15:15:08 | jaypipes | yeah, I understand that now. Why don't we just list() that instead of returning a generator? | |
| 15:15:26 | mdbooth | Well that would still be a race, just a shorter one | |