| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-19 | |||
| 17:33:40 | lennyb | Hi, my nova instance got stuck in 'deleting' when I try to delete instance that failed to be deployed via ironic n-cell-child.service.log | |
| 17:33:41 | lennyb | http://paste.openstack.org/show/704774/ | |
| 17:41:16 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Mention (no) granular support for image traits https://review.openstack.org/554305 | |
| 17:42:33 | efried | jaypipes, mriedem: ^ | |
| 17:46:23 | oomichi | melwitt: cdent: hi, do we have an approval at the PTG for https://blueprints.launchpad.net/nova/+spec/placement-extract ? | |
| 17:46:27 | jaypipes | dansmith, mriedem: would you mind looking at https://review.openstack.org/#/c/534339/ please and letting me know if you agree with efried? I'm thinking I agree with him and if so, will abandon that and the following patch to it. | |
| 17:46:54 | cdent | oomichi: the decision at ptg was to make progress in rocky, but not complete it, and not as a priority | |
| 17:47:57 | oomichi | cdent: I see, thanks. I just wanted to see the bp as approved | |
| 17:48:54 | cdent | oomichi: I didn't actually create the bp until late last week, because efried suggested it would be useful. We hadn't really declared "let's have one" but it does seem like a good idea. | |
| 17:50:01 | melwitt | oomichi, cdent: yeah, seems useful to have it to track the work. I'll approve it | |
| 17:50:07 | oomichi | cdent: yeah, bp is good for tracking patches | |
| 17:50:14 | oomichi | melwitt: cool, thanks :) | |
| 17:55:23 | efried | cdent, jaypipes: (how) does forbidden traits play into glance image traits? | |
| 17:57:01 | cdent | efried: i would guess the same as flavors? I haven't put that much thought into that side of things, just the api side | |
| 17:57:30 | jaypipes | efried: I would expect that the format of specifying a forbidden trait would be the same. I would also expect that if there was a conflict between flavor's forbidden traits and image required traits (and vice versa), that we would bomb out. | |
| 17:57:53 | efried | cdent: Oh, right (rereading the spec now), we haven't even tried to talk about how forbidden traits will be expressed in flavors, have we? | |
| 17:58:07 | efried | Is that simply not planned for R? | |
| 17:59:02 | jaypipes | efried: I would have thought that s//required/forbidden in the extra spec value field would be how we'd do it. | |
| 17:59:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: slow live-migration to ensure network is ready https://review.openstack.org/497457 | |
| 18:02:38 | openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Fix typos in release notes https://review.openstack.org/554175 | |
| 18:06:16 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553817 | |
| 18:09:05 | melwitt | dansmith, mriedem: looks like we have an interesting bug here during an upgrade ocata -> pike where while reading a service record, if it doesn't have a uuid, we generate one and try to save it back to the object, but that fails with "Can't upgrade a READER transaction to a WRITE" because a read will start off with @pick_context_manager_reader and we can't upgrade to a writer in the middle of the transaction https://bugs.launchpad.net/n | |
| 18:09:05 | melwitt | ova/+bug/1746509/comments/4 | |
| 18:09:14 | melwitt | dangit | |
| 18:09:16 | melwitt | https://bugs.launchpad.net/nova/+bug/1746509/comments/4 | |
| 18:09:17 | openstack | Launchpad bug 1722404 in OpenStack Compute (nova) ocata "duplicate for #1746509 Database transactions can fail with "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" because of scatter_gather_cells" [Undecided,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 18:10:41 | dansmith | hmm, I'm not sure why it'd be different for scatter/gather | |
| 18:10:59 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553817 | |
| 18:11:22 | melwitt | I don't think it's different. I think this is a separate different bug, but was highlighting the comment where someone explained what was going on | |
| 18:11:54 | dansmith | yeah, that explanation makes sense but I'm not sure why we weren't hitting that immediately is what I mean | |
| 18:11:59 | melwitt | I think we fixed the scatter gather problem but this service record thing is separate and different | |
| 18:12:03 | melwitt | oh | |
| 18:13:22 | melwitt | I assume we were ... trying to think if this scenario is covered by grenade or how it wouldn't be | |
| 18:13:59 | dansmith | well, it should have manifested even in functional tests I would think | |
| 18:14:04 | dansmith | we must be in a nested call where we're already in a reader transaction and we call to do a write, right? | |
| 18:14:19 | melwitt | yeah, that would be the condition for that error being raised | |
| 18:14:21 | dansmith | blanking out the context dumps that so we get a new transaction for the write | |
| 18:18:08 | melwitt | you mean the way that the commenter patched nova/objects/service.py? yeah | |
| 18:18:47 | dansmith | yeah they used a new context | |
| 18:19:24 | melwitt | yeah | |
| 18:31:03 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553818 | |
| 18:35:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | |
| 18:37:34 | mriedem | dansmith: http://lists.openstack.org/pipermail/openstack-dev/2018-March/128487.html for the setuptools thing you were talking about before | |
| 18:37:37 | mriedem | pkg_version | |
| 18:37:57 | dansmith | ah cool | |
| 18:37:59 | dansmith | migi: ^ | |
| 18:38:30 | dansmith | mriedem: did you see my comment on that patch for the obj_relationships thing? | |
| 18:39:13 | mriedem | not yet | |
| 18:44:31 | melwitt | dansmith: from what I can tell, we only have unit test coverage for the uuid generation for service where things are mocked (and would anyway be a write to begin with), or calling _from_db_object selectively https://github.com/openstack/nova/blob/master/nova/tests/unit/objects/test_service.py#L145 | |
| 18:44:54 | melwitt | to hit this, we'd have to have a functional test scenario where we read an already existing service record that has no uuid | |
| 18:45:03 | melwitt | I can try writing one | |
| 18:47:24 | dansmith | hmm, I thought we did | |
| 18:47:30 | dansmith | because that was mriedem | |
| 18:47:36 | dansmith | and he looooves functional tests | |
| 18:47:56 | dansmith | I would also have expected to hear more noise about this | |
| 18:48:20 | dansmith | like, wouldn't the first compute startup after that patch merged cause nova-compute to not update its service record? | |
| 18:49:43 | mriedem | it was dtp with me helping, but it's a pretty established pattern, | |
| 18:49:53 | mriedem | i.e. he copied it from compute_nodes.uuid, and aggregates.uuid | |
| 18:49:57 | dansmith | yeah | |
| 18:50:04 | dansmith | was just lookin gup the commit | |
| 18:50:12 | mriedem | also https://github.com/dpeschman | |
| 18:50:17 | mriedem | https://avatars3.githubusercontent.com/u/145098?s=400&v=4 | |
| 18:50:26 | mriedem | i would like to know what's going on there | |
| 18:50:36 | melwitt | lol | |
| 18:51:59 | dansmith | even still, | |
| 18:52:11 | dansmith | we're out of the db context at that point, so I'm not sure why this is happening anyway | |
| 18:52:34 | dansmith | out of the pick_context_manager I mean | |
| 19:11:51 | openstackgerrit | Merged openstack/nova-specs master: Update placement aggregates spec to clarify generation handling https://review.openstack.org/548237 | |
| 19:23:12 | openstackgerrit | Dan Smith proposed openstack/nova master: Add AggregateList.get_by_metadata() query method https://review.openstack.org/544728 | |
| 19:23:13 | openstackgerrit | Dan Smith proposed openstack/nova master: Add aggregates list to Destination object https://review.openstack.org/544729 | |
| 19:23:13 | openstackgerrit | Dan Smith proposed openstack/nova master: Add request filter functionality to scheduler https://review.openstack.org/544730 | |
| 19:23:14 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990 | |
| 19:23:14 | openstackgerrit | Dan Smith proposed openstack/nova master: [WIP] Add require_tenant_aggregate request filter https://review.openstack.org/545002 | |
| 19:23:15 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282 | |
| 19:35:52 | mriedem | debugging weirdness during live migratoin would be a lot easier if we had thread ids in the logs... | |
| 19:40:47 | mriedem | sean-k-mooney: do you know if unbinding a port makes it automatically go to status=DOWN? | |
| 19:44:19 | mriedem | sean-k-mooney: https://review.openstack.org/#/c/434870/11/nova/compute/manager.py@1126 | |
| 19:44:55 | mriedem | seeing issues during live migration where we handle the migration completed lifecycle event from the hypervisor and start unbinding the port while we're also going through _post_live_migration | |
| 19:45:02 | mriedem | we blow up trying to unplug an unbound VIF | |
| 19:45:18 | sean-k-mooney[m] | Im not sure. The status of an unbound port does not really have a meaning as it is effectively just a db entry when unbound. E. G. It dose not exist on the datapath when unbound. | |
| 19:45:34 | mriedem | i ask because of https://github.com/openstack/neutron/blob/f2dced777acb7d87dc7048c3efec59dadf37930d/neutron/notifiers/nova.py#L186 | |
| 19:45:50 | mriedem | i'm seeing mysterious network-vif-plugged and network-vif-unplugged events around the time of the failure | |
| 19:46:00 | mriedem | but they are coming before we unplug vifs on the source host | |
| 19:47:49 | mriedem | might be getting network-vif-plugged events on the source host from when they were plugged on the dest host in pre_live_migration, | |
| 19:48:00 | mriedem | the events are going to be routed to where the instance currrently lives, which is the source host until we switch | |
| 19:48:57 | mriedem | then we live migrate the guest, | |
| 19:49:10 | mriedem | get the 'migration completed' lifecycle event from the hypervisor and start unbinding the port, | |
| 19:49:11 | dansmith | mriedem: once there is a migration record, they'll be routed to both hosts | |
| 19:49:50 | sean-k-mooney[m] | Well in the old workflow we don't unbind the port. We just update the host in the binding details. Yes i think the vif plugged events are from when the destination vif is plugged | |
| 19:49:51 | mriedem | ok, i was only checking the source host logs, can check the dest host logs | |
| 19:50:18 | mriedem | sean-k-mooney[m]: this is the patch causing problems right now https://review.openstack.org/#/c/434870/ | |
| 19:50:31 | mriedem | the thing that tried to do the early switch once were in post-copy | |
| 19:50:37 | mriedem | *once we were in | |
| 19:50:56 | jroll | jaypipes: dansmith: this is roughly what I recall from PTG about ironic RT stuff, does it make sense to you? https://review.openstack.org/#/c/545479/ | |
| 19:53:33 | mriedem | dansmith: hmm, not seeing the same event routed to both hosts | |
| 19:53:48 | mriedem | but we should have a migration record at this point because i see on the dest host logs that we've plugged the vifs there in pre_live_migration | |
| 19:55:30 | dansmith | mriedem: hang on I'm piecing this back together in my head just a sec | |
| 19:55:36 | dansmith | jroll: ack, will look ater | |
| 19:56:00 | dansmith | mriedem: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4547-L4559 | |
| 19:56:01 | jroll | thanks | |