| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-20 | |||
| 17:17:52 | mriedem | if so, we don't know how to translate that to an os-vif object to plug the vif on the dest host | |
| 17:17:54 | mriedem | and it blows up | |
| 17:18:00 | jroll | mriedem: eh, that's a bigger timesink that I'm not sure I want to do in the middle of that loop... though would help in the case of a whole rack getting unplugged or something | |
| 17:18:27 | mriedem | mdbooth: semi related https://review.openstack.org/#/c/603844/ | |
| 17:18:53 | mdbooth | mriedem: Well it remains bound on the source, so it looks normal I guess? | |
| 17:18:54 | mriedem | mdbooth: if the vif_type in the port binding is busted, you might have to detach and re-attach the port to the server to fix it | |
| 17:19:09 | mriedem | mdbooth: check the port binding details on the port in the neutron api | |
| 17:19:16 | mriedem | you probably have to be admin to get that field | |
| 17:19:35 | mriedem | binding:vif_type is what you're looking for | |
| 17:19:42 | mdbooth | It's 'ovs' | |
| 17:19:53 | mriedem | oh huh | |
| 17:20:03 | mriedem | well actually, | |
| 17:20:12 | mriedem | is this rocky? | |
| 17:20:15 | mdbooth | mriedem: This is a dark art to me, btw :) | |
| 17:20:17 | mdbooth | devstack master | |
| 17:20:36 | mriedem | the port binding for the dest host will be on GET /ports/{port_id}/bindings/{dest_host} | |
| 17:20:47 | mriedem | the port binding details you're looking at on the port are likely the source host | |
| 17:20:53 | mriedem | which is the 'active' binding | |
| 17:20:58 | mriedem | the dest host binding is the inactive one | |
| 17:21:02 | mriedem | and it's what you're trying to plug | |
| 17:21:10 | mdbooth | Ah, ok. | |
| 17:21:19 | mdbooth | No cli for that? | |
| 17:21:25 | sean-k-mooney | mdbooth: it was in the spec but no | |
| 17:21:26 | mriedem | unfortunately there is no api ref for the port bindings api yet | |
| 17:22:01 | mdbooth | I'm assuming if I look at the LM code in nova we're failing to clean that up | |
| 17:23:01 | sean-k-mooney | mdbooth: to clean up a host binding? | |
| 17:23:51 | mriedem | _rollback_live_migration should delete inactive dest host port bindings | |
| 17:23:53 | sean-k-mooney | mdbooth: i had tought we had code for that | |
| 17:24:21 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958 | |
| 17:24:39 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7010 | |
| 17:25:12 | mriedem | if we failed to cleanup the dest host port bindings where should be an error in the compute logs | |
| 17:27:41 | mdbooth | Hmm, nothing obvious | |
| 17:28:08 | mdbooth | Anyway, I'm going to have to shoot. I'll investigate properly in the morning, was just sharing in case sean-k-mooney had immediate insight. | |
| 17:28:25 | mdbooth | Thanks for the help. | |
| 17:30:55 | mriedem | dansmith: commented https://review.openstack.org/#/c/635147/ | |
| 17:38:04 | sean-k-mooney | mdbooth i know that i have seen case where cleaning up the volume on a roleback fails and we raise an excpetion and dont then rollback the networking | |
| 17:39:29 | sean-k-mooney | mdbooth: my guess is that is the most likely cause. for example if self.compute_rpcapi.remove_volume_connection rased an exception we would never call self.network_api.setup_networks_on_host | |
| 17:40:43 | sean-k-mooney | mdbooth: we probaly should split _rollback_live_migration into several smaller fucntion and call them with a set fo try finally statement to ensure that all resouces that can be rolled back are | |
| 17:42:25 | sean-k-mooney | mdbooth: ill be fixing stuff arount this code for something else next sprint so i might look at doing that change then. | |
| 17:43:30 | jaypipes | gibi: still around? | |
| 17:44:00 | jaypipes | gibi: on https://review.openstack.org/#/c/616239/ did we decide not to change anything for the group_policy_isolate=true? | |
| 18:11:07 | artom | Resizing to the same host is expected to work, right? | |
| 18:11:10 | openstackgerrit | Dan Smith proposed openstack/nova master: API microversion 2.69: Handles Down Cells Documentation https://review.openstack.org/635147 | |
| 18:11:34 | artom | I'm debugging an internal thing, and it looks like we're still asking Neutron to update the port bindings, even though nothing's changed. | |
| 18:12:11 | artom | Bleargh no, it's the revert. | |
| 18:12:14 | artom | Never mind. | |
| 18:12:21 | artom | Thanks for being my rubber duck. | |
| 18:13:32 | sean-k-mooney | same host resize needs to be enable in the nova config i think but yes it should work | |
| 18:15:16 | sean-k-mooney | artom: by the way we do need to rebind on the neutron side as the operator could have changed the security group driver for instance so how we attach the instance to the netowrk backend could change | |
| 18:15:51 | sean-k-mooney | artom: this can happen for example if you did and inplace upgrade and then resize the vm | |
| 18:16:09 | artom | sean-k-mooney, aha, ok | |
| 18:16:11 | sean-k-mooney | normally the changes would get picked up on the next hard reboot | |
| 18:16:56 | sean-k-mooney | artom: but ya were you actully hittin an issue? | |
| 18:17:10 | artom | sean-k-mooney, not 100% sure yet, but looks like it's Neutron | |
| 18:17:23 | artom | We ask for a thing, and timeout waiting for it | |
| 18:18:16 | artom | Grr, and I can't find the corresponding Neutron request | |
| 18:18:35 | sean-k-mooney | oh is this the virtual interface thing we were talking about downstream earlier | |
| 18:19:48 | artom | Yeah | |
| 18:20:29 | sean-k-mooney | artom: as i mentioned to mdbooth eariler if any of the storage cleanup code raises an exception in _rollback_live_migration its possible today that we will never call self.network_api.setup_network_on_host to rollback the network | |
| 18:20:51 | sean-k-mooney | so if there is an exption earilier in the function https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958 will not be called | |
| 18:21:01 | artom | sean-k-mooney, no, I think we're reaching that part | |
| 18:21:10 | artom | As I said, I see Nova making the Neutron request | |
| 18:21:39 | artom | But debug's not enabled, so I can't find that request arriving on the Neutron side | |
| 18:21:54 | artom | I mean, maybe it doesn't? | |
| 18:22:09 | sean-k-mooney | you shoudl see it in the neutron api log without debugging | |
| 18:22:10 | artom | Actually, debug *is* enabled | |
| 18:22:58 | sean-k-mooney | you may not see al of the intermediate steps but the post to the port binding endoint should be there | |
| 18:23:32 | artom | It'll log the HTTP method, right? | |
| 18:23:45 | artom | I'm casting a wide net and straight up grep -E '2019-02-14 18:50:5.*(PUT|POST)' -R `find . -wholename '*neutron/*.log'` | |
| 18:23:46 | sean-k-mooney | ya and it should have the respocne code | |
| 18:23:51 | artom | Nothing. | |
| 18:25:23 | artom | That's weird though, 'cuz Nova would log a timeout | |
| 18:25:44 | sean-k-mooney | so it timed out waiting for the network-vif-plugged event | |
| 18:26:12 | sean-k-mooney | whcich got raised as a VirtualInterfaceCreateExcetiopn. | |
| 18:27:00 | sean-k-mooney | the logs i have did not containt the neutron server logs | |
| 18:27:30 | artom | sean-k-mooney, that might be why | |
| 18:46:52 | imacdonn | Is it just me, or does this releasenote seem incomprehensive? I'm left wondering whether or not I should remove the consoleauth service, and under what circumstances the workaround is required. "A new check is added to the nova-status upgrade check CLI to check for use of the nova-consoleauth service to warn and provide additional instructions to set [workarounds]enable_consoleauth = True while performing a live/rolling upgrade." | |
| 18:59:41 | imacdonn | I think it should say something like "The consoleauth service is deprecated, and should be disabled/removed, however, if there is a requirement to maintain support for existing console sessions through a live/rolling upgrade......" | |
| 19:00:55 | melwitt | consoleauth strikes again | |
| 19:01:30 | melwitt | imacdonn: thanks for the feedback. I actually tried to explain it better in the latest updates to the releasenote and still missed the mark. I can propose another change | |
| 19:01:57 | melwitt | you definitely need to enable the workaround if you're doing a live upgrade | |
| 19:02:27 | imacdonn | yeah, seems we've been here before ;) At least removing the service seems to actually not generally break things this time ;) | |
| 19:02:35 | aspiers | melwitt: are release notes only for operator- / user-facing changes? | |
| 19:02:54 | aspiers | just wondering if I should add one for this new driver-capabilities-to-traits patch | |
| 19:06:42 | imacdonn | heh | |
| 19:07:07 | melwitt | aspiers: yeah, anything users should know about a change. I don't think I've seen the driver capabilities to traits thing yet | |
| 19:07:29 | aspiers | melwitt: https://review.openstack.org/#/c/538498/12 | |
| 19:07:40 | aspiers | melwitt: it's my rebase of an old patch from mriedem | |
| 19:07:45 | aspiers | I'm currently fixing the test failures | |
| 19:09:31 | imacdonn | aspiers: I guess my question (if I mattered) would be: who needs to know about this ? | |
| 19:09:45 | melwitt | aspiers: ok, I see. I think that would use a release note to let users know that driver capabilities will show up as custom traits in the placement API | |
| 19:10:18 | aspiers | imacdonn: my question too ;-) | |
| 19:10:24 | aspiers | melwitt: OK thanks! | |
| 19:12:26 | melwitt | aspiers: as for who, I don't know exactly, but it's the kind of thing I could see a cloud monitoring system looking at, what driver capabilities exist on each compute host | |
| 19:12:51 | aspiers | melwitt: yup, makes sense | |
| 19:13:22 | imacdonn | if there's a change to the way things appear through any published interface, I'd say it definitely should be a release-note. | |
| 19:13:52 | aspiers | yeah, I suppose just new traits appearing would be sufficient to justify documenting | |
| 19:18:15 | imacdonn | tangentially related (to release notes), I found a typo in another one. I might submit a fix a bit later, if I get time, but if someone else wants to grab it, that'd be cool too. https://bugs.launchpad.net/nova/+bug/1816831 | |
| 19:18:15 | openstack | Launchpad bug 1816831 in OpenStack Compute (nova) "DOC: typo in add_initial_allocation_ratio releasenote" [Undecided,New] | |
| 19:20:52 | melwitt | thanks | |