| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-20 | |||
| 17:23:01 | sean-k-mooney | mdbooth: to clean up a host binding? | |
| 17:23:51 | mriedem | _rollback_live_migration should delete inactive dest host port bindings | |
| 17:23:53 | sean-k-mooney | mdbooth: i had tought we had code for that | |
| 17:24:21 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958 | |
| 17:24:39 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7010 | |
| 17:25:12 | mriedem | if we failed to cleanup the dest host port bindings where should be an error in the compute logs | |
| 17:27:41 | mdbooth | Hmm, nothing obvious | |
| 17:28:08 | mdbooth | Anyway, I'm going to have to shoot. I'll investigate properly in the morning, was just sharing in case sean-k-mooney had immediate insight. | |
| 17:28:25 | mdbooth | Thanks for the help. | |
| 17:30:55 | mriedem | dansmith: commented https://review.openstack.org/#/c/635147/ | |
| 17:38:04 | sean-k-mooney | mdbooth i know that i have seen case where cleaning up the volume on a roleback fails and we raise an excpetion and dont then rollback the networking | |
| 17:39:29 | sean-k-mooney | mdbooth: my guess is that is the most likely cause. for example if self.compute_rpcapi.remove_volume_connection rased an exception we would never call self.network_api.setup_networks_on_host | |
| 17:40:43 | sean-k-mooney | mdbooth: we probaly should split _rollback_live_migration into several smaller fucntion and call them with a set fo try finally statement to ensure that all resouces that can be rolled back are | |
| 17:42:25 | sean-k-mooney | mdbooth: ill be fixing stuff arount this code for something else next sprint so i might look at doing that change then. | |
| 17:43:30 | jaypipes | gibi: still around? | |
| 17:44:00 | jaypipes | gibi: on https://review.openstack.org/#/c/616239/ did we decide not to change anything for the group_policy_isolate=true? | |
| 18:11:07 | artom | Resizing to the same host is expected to work, right? | |
| 18:11:10 | openstackgerrit | Dan Smith proposed openstack/nova master: API microversion 2.69: Handles Down Cells Documentation https://review.openstack.org/635147 | |
| 18:11:34 | artom | I'm debugging an internal thing, and it looks like we're still asking Neutron to update the port bindings, even though nothing's changed. | |
| 18:12:11 | artom | Bleargh no, it's the revert. | |
| 18:12:14 | artom | Never mind. | |
| 18:12:21 | artom | Thanks for being my rubber duck. | |
| 18:13:32 | sean-k-mooney | same host resize needs to be enable in the nova config i think but yes it should work | |
| 18:15:16 | sean-k-mooney | artom: by the way we do need to rebind on the neutron side as the operator could have changed the security group driver for instance so how we attach the instance to the netowrk backend could change | |
| 18:15:51 | sean-k-mooney | artom: this can happen for example if you did and inplace upgrade and then resize the vm | |
| 18:16:09 | artom | sean-k-mooney, aha, ok | |
| 18:16:11 | sean-k-mooney | normally the changes would get picked up on the next hard reboot | |
| 18:16:56 | sean-k-mooney | artom: but ya were you actully hittin an issue? | |
| 18:17:10 | artom | sean-k-mooney, not 100% sure yet, but looks like it's Neutron | |
| 18:17:23 | artom | We ask for a thing, and timeout waiting for it | |
| 18:18:16 | artom | Grr, and I can't find the corresponding Neutron request | |
| 18:18:35 | sean-k-mooney | oh is this the virtual interface thing we were talking about downstream earlier | |
| 18:19:48 | artom | Yeah | |
| 18:20:29 | sean-k-mooney | artom: as i mentioned to mdbooth eariler if any of the storage cleanup code raises an exception in _rollback_live_migration its possible today that we will never call self.network_api.setup_network_on_host to rollback the network | |
| 18:20:51 | sean-k-mooney | so if there is an exption earilier in the function https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6958 will not be called | |
| 18:21:01 | artom | sean-k-mooney, no, I think we're reaching that part | |
| 18:21:10 | artom | As I said, I see Nova making the Neutron request | |
| 18:21:39 | artom | But debug's not enabled, so I can't find that request arriving on the Neutron side | |
| 18:21:54 | artom | I mean, maybe it doesn't? | |
| 18:22:09 | sean-k-mooney | you shoudl see it in the neutron api log without debugging | |
| 18:22:10 | artom | Actually, debug *is* enabled | |
| 18:22:58 | sean-k-mooney | you may not see al of the intermediate steps but the post to the port binding endoint should be there | |
| 18:23:32 | artom | It'll log the HTTP method, right? | |
| 18:23:45 | artom | I'm casting a wide net and straight up grep -E '2019-02-14 18:50:5.*(PUT|POST)' -R `find . -wholename '*neutron/*.log'` | |
| 18:23:46 | sean-k-mooney | ya and it should have the respocne code | |
| 18:23:51 | artom | Nothing. | |
| 18:25:23 | artom | That's weird though, 'cuz Nova would log a timeout | |
| 18:25:44 | sean-k-mooney | so it timed out waiting for the network-vif-plugged event | |
| 18:26:12 | sean-k-mooney | whcich got raised as a VirtualInterfaceCreateExcetiopn. | |
| 18:27:00 | sean-k-mooney | the logs i have did not containt the neutron server logs | |
| 18:27:30 | artom | sean-k-mooney, that might be why | |
| 18:46:52 | imacdonn | Is it just me, or does this releasenote seem incomprehensive? I'm left wondering whether or not I should remove the consoleauth service, and under what circumstances the workaround is required. "A new check is added to the nova-status upgrade check CLI to check for use of the nova-consoleauth service to warn and provide additional instructions to set [workarounds]enable_consoleauth = True while performing a live/rolling upgrade." | |
| 18:59:41 | imacdonn | I think it should say something like "The consoleauth service is deprecated, and should be disabled/removed, however, if there is a requirement to maintain support for existing console sessions through a live/rolling upgrade......" | |
| 19:00:55 | melwitt | consoleauth strikes again | |
| 19:01:30 | melwitt | imacdonn: thanks for the feedback. I actually tried to explain it better in the latest updates to the releasenote and still missed the mark. I can propose another change | |
| 19:01:57 | melwitt | you definitely need to enable the workaround if you're doing a live upgrade | |
| 19:02:27 | imacdonn | yeah, seems we've been here before ;) At least removing the service seems to actually not generally break things this time ;) | |
| 19:02:35 | aspiers | melwitt: are release notes only for operator- / user-facing changes? | |
| 19:02:54 | aspiers | just wondering if I should add one for this new driver-capabilities-to-traits patch | |
| 19:06:42 | imacdonn | heh | |
| 19:07:07 | melwitt | aspiers: yeah, anything users should know about a change. I don't think I've seen the driver capabilities to traits thing yet | |
| 19:07:29 | aspiers | melwitt: https://review.openstack.org/#/c/538498/12 | |
| 19:07:40 | aspiers | melwitt: it's my rebase of an old patch from mriedem | |
| 19:07:45 | aspiers | I'm currently fixing the test failures | |
| 19:09:31 | imacdonn | aspiers: I guess my question (if I mattered) would be: who needs to know about this ? | |
| 19:09:45 | melwitt | aspiers: ok, I see. I think that would use a release note to let users know that driver capabilities will show up as custom traits in the placement API | |
| 19:10:18 | aspiers | imacdonn: my question too ;-) | |
| 19:10:24 | aspiers | melwitt: OK thanks! | |
| 19:12:26 | melwitt | aspiers: as for who, I don't know exactly, but it's the kind of thing I could see a cloud monitoring system looking at, what driver capabilities exist on each compute host | |
| 19:12:51 | aspiers | melwitt: yup, makes sense | |
| 19:13:22 | imacdonn | if there's a change to the way things appear through any published interface, I'd say it definitely should be a release-note. | |
| 19:13:52 | aspiers | yeah, I suppose just new traits appearing would be sufficient to justify documenting | |
| 19:18:15 | imacdonn | tangentially related (to release notes), I found a typo in another one. I might submit a fix a bit later, if I get time, but if someone else wants to grab it, that'd be cool too. https://bugs.launchpad.net/nova/+bug/1816831 | |
| 19:18:15 | openstack | Launchpad bug 1816831 in OpenStack Compute (nova) "DOC: typo in add_initial_allocation_ratio releasenote" [Undecided,New] | |
| 19:20:52 | melwitt | thanks | |
| 19:33:23 | mriedem | dansmith: +2 on that docs change | |
| 19:33:58 | dansmith | mriedem: +W | |
| 19:34:05 | mriedem | let us never speak of this again | |
| 19:34:25 | dansmith | I looked a the best practices one a bit, found one typo, but will look again when it gets respun | |
| 19:35:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix typo in initial_disk_allocation_ratio release note https://review.openstack.org/638245 | |
| 19:35:44 | mriedem | imacdonn: done - thanks for reporting that ^ | |
| 19:36:10 | imacdonn | mriedem: thanks! | |
| 19:37:12 | dansmith | +W within one minute | |
| 19:37:46 | melwitt | whew, what a sense of accomplishment | |
| 19:39:28 | aspiers | almost as towering an achievement as https://review.openstack.org/#/c/609777/ | |
| 19:39:45 | melwitt | lol | |
| 19:39:49 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add support for microversion 2.70 - expose device tags https://review.openstack.org/636779 | |
| 19:39:54 | aspiers | melwitt: my finest hour | |
| 19:46:04 | artom | aspiers, I'm grateful that your around | |
| 19:48:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.openstack.org/635308 | |
| 19:52:41 | imacdonn | artom: I see what you did there ;) | |
| 20:02:27 | artom | imacdonn, I no you did. | |
| 20:02:36 | imacdonn | :) | |
| 20:03:12 | aspiers | artom: stop teasing, its cruel | |
| 20:18:01 | aspiers | ;) | |
| 20:52:41 | mriedem | low-hanging-fruit docs bug if someone wants it https://bugs.launchpad.net/nova/+bug/1816859 | |
| 20:52:42 | openstack | Launchpad bug 1816859 in OpenStack Compute (nova) "Server concepts in nova - automatic resize confirm is wrong in docs" [Medium,Triaged] | |
| 20:54:30 | openstackgerrit | Adam Spiers proposed openstack/nova master: Convert driver supported capabilities to compute node provider traits https://review.openstack.org/538498 | |
| 20:59:09 | openstackgerrit | Adam Spiers proposed openstack/nova master: Convert driver supported capabilities to compute node provider traits https://review.openstack.org/538498 | |
| 21:00:01 | aspiers | efried_pto, mriedem, melwitt: I think this is ready for serious consideration now ^^^ | |