| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-15 | |||
| 22:44:19 | belmoreira | my previous point was that if for a specific use case we need to create aggregates/add nodes it will be very difficult to manage this at scale | |
| 22:45:14 | melwitt | mriedem: replied on https://review.openstack.org/#/c/340614 about instance.host = None + vm_state = ERROR | |
| 22:46:13 | belmoreira | especially because aggregates and cells need to be in sync. Missing a node in this mapping could mean that it will not be used even if it's in the cell | |
| 22:47:05 | openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507 | |
| 22:48:57 | mriedem | edmondsw: efried: done https://review.openstack.org/#/c/545111/ | |
| 22:49:06 | efried | mriedem: Thanks! | |
| 22:53:32 | mriedem | melwitt: we expect that in a *very specific scenario* | |
| 22:53:59 | mriedem | the commit message makes it sound like the instance is ever only in error state because of a failed build, which is not the case | |
| 22:56:30 | melwitt | okay, so the words should be "we can expect" instead of "we expect" | |
| 22:56:54 | mriedem | let me get cochran on the horn | |
| 22:57:55 | melwitt | "on the horn" means phone? that's new to me | |
| 22:58:04 | mriedem | yes | |
| 22:59:04 | mriedem | did you get the cochran reference at least? | |
| 22:59:27 | melwitt | like johnny cochran? | |
| 22:59:30 | mriedem | yes | |
| 22:59:37 | melwitt | yeah | |
| 23:00:20 | mriedem | how about just saying, "If the instance is in ERROR because of a failed build" | |
| 23:00:49 | mriedem | my council informs me that would be satisfactory | |
| 23:01:28 | melwitt | thank you, council people | |
| 23:08:39 | openstack | Launchpad bug 1749838 in OpenStack Compute (nova) "Rescheduled instace with pre-existing port fails with PortInUse exception" [Undecided,New] | |
| 23:08:39 | imacdonn | mriedem: I reproduced the PortInUse thing on Pike, and also confirmed it only happens when using a pre-existing port - created https://bugs.launchpad.net/nova/+bug/1749838 | |
| 23:10:48 | mriedem | ok | |
| 23:12:11 | mriedem | imacdonn: this is latest stable/pike? | |
| 23:12:44 | imacdonn | It's RDO - nova 16.0.3 | |
| 23:12:55 | mriedem | ok so port.device_id is set | |
| 23:13:01 | mriedem | is what it's failing on | |
| 23:13:39 | mriedem | when we unbind the port before rescheduling, we should wipe that out | |
| 23:13:40 | mriedem | port_req_body = {'port': {'device_id': '', 'device_owner': ''}} | |
| 23:14:06 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/network/neutronv2/api.py#L511 | |
| 23:14:21 | mriedem | do you see "Unable to clear device ID" in the logs? | |
| 23:14:23 | mriedem | on the first host? | |
| 23:14:39 | imacdonn | will look .. I did attach debug lots from both nodes to the bug just now | |
| 23:14:43 | imacdonn | logs* | |
| 23:15:55 | imacdonn | I don't see that message (grep for "clear" only finds a couple of config options) | |
| 23:16:28 | mriedem | i feel like i had a patch that added debug logging in this code for when we tore down, but i probably abandoned it | |
| 23:17:15 | mriedem | imacdonn: oh i bet this is the fix you need https://review.openstack.org/#/c/520248/ | |
| 23:18:01 | mriedem | in the case of nova creating a port, | |
| 23:18:12 | mriedem | it doesn't fail because nova orphans the port created from the first host, and creates a new port when going through the 2nd host | |
| 23:18:24 | mriedem | so you end up with 2 ports for the instance that nova created even though you're only using 1 | |
| 23:18:42 | mriedem | in the case that you bring a port, nova doesn't unbind it before rescheduling, and that's why we fail to use it on the 2nd host | |
| 23:18:52 | imacdonn | seems plausible ... looks like a one-line change, so I can try to drop it in | |
| 23:18:55 | mriedem | imacdonn: you could try applying that patch and see if it resolves it | |
| 23:35:27 | imacdonn | mriedem: yup, that fixed it - thanks! | |
| 23:46:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608 | |
| 23:46:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 23:54:52 | cfriesen | does anyone know offhand what the rules are for using instance.info_cache.network_info vs calling self.network_api.get_instance_nw_info(context, instance) ? | |
| #openstack-nova - 2018-02-16 | |||
| 00:03:12 | tonyb | mriedem: any chnace you can take a quick look at: https://review.openstack.org/#/c/543348 ? | |
| 00:09:21 | openstackgerrit | Nicolas Bock proposed openstack/nova stable/pike: Fix SUSE Install Guide: Placement port https://review.openstack.org/545167 | |
| 00:11:42 | melwitt | mnaser, mriedem: I'm going through your comments on https://review.openstack.org/#/c/340614/18/nova/compute/api.py@2055 and I'm not seeing what the 'available' volume scenario is ... | |
| 00:13:00 | melwitt | that is, even back in liberty, if the instance failed on the host, _cleanup_volumes in the compute manager would have called volume_api.delete, so maybe that's why the detach would fail in the compute api local delete ... but that doesn't add up with having a volume left in 'available' state | |
| 00:13:58 | mnaser | melwitt: in that case you are right there wouldn’t be any possible scenario where you would end up in an available volume if it failed in compute | |
| 00:14:06 | melwitt | so I'm not really sure what case the new except block is handling | |
| 00:14:39 | melwitt | (I didn't add that part but back when this was going on in liberty, an operator we were working with added that saying they ran into it when they were testing the patch) | |
| 00:15:30 | melwitt | I'm trying to figure out whether to nix it from this patch or if there's some scenario I'm missing where we need to handle it like this | |
| 00:16:04 | mnaser | melwitt: I just got home so I’m on mobile but can you boot from volume with an existing volume that’s also delete on terminate? | |
| 00:16:32 | mnaser | That would be a scenario where you’d want us to delete it, assuming scheduling failed and we want to follow through and delete the volume as requested | |
| 00:16:47 | mnaser | Though that might be super confusing to the user and unexpected | |
| 00:20:03 | cfriesen | mnaser: in that case wouldn't the instance sit around in an ERROR state with the volume still attached? | |
| 00:20:26 | cfriesen | ie the volume should stay attached until the instance is deleted | |
| 00:21:34 | melwitt | mnaser: cfriesen it was that way until very recently https://review.openstack.org/#/c/528385/ | |
| 00:23:04 | mnaser | Will the volume ever end up on attached state? I think it would be in attaching because it never reached the compute note to finish the attach | |
| 00:27:34 | openstackgerrit | Merged openstack/nova stable/queens: Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/542738 | |
| 00:27:40 | openstackgerrit | Merged openstack/nova stable/queens: Don't JSON encode instance_info.traits for ironic https://review.openstack.org/545037 | |
| 00:27:48 | openstackgerrit | Merged openstack/nova stable/queens: Use correct arguments in task inits https://review.openstack.org/544109 | |
| 00:28:00 | openstackgerrit | Merged openstack/nova stable/queens: Update UPPER_CONSTRAINTS_FILE for stable/queens https://review.openstack.org/542657 | |
| 00:28:05 | melwitt | mnaser: yeah, you're right, in the scenario you're describing, BFV with already existing volume + delete_on_termination, if scheduling fails, the volume would never have been attached | |
| 00:28:58 | melwitt | so it would be available, so detach would fail, so we would want to delete the volume. | |
| 00:29:37 | melwitt | er, detach might just do a no-op if available, but I'm not sure on that | |
| 00:31:37 | mnaser | melwitt: wouldn’t it actually be in “attaching” state because the API layer reserves the volume | |
| 00:32:00 | melwitt | mnaser: oh, yeah good point. it will be whatever the reserve did | |
| 00:32:52 | mnaser | melwitt: yeah so it would have an attachment or reservation depending on flow, same issue we’re trying to resolve here | |
| 00:34:06 | mnaser | melwitt: I do think the extra try except added in this patch is useless if detach is noop | |
| 00:34:39 | mnaser | If detach is noop, nothing will happen. If it fails because the volume doesn’t exist, then we won’t try to delete it anyways | |
| 00:34:53 | melwitt | yeah. that's where I'm at too | |
| 00:35:22 | mnaser | Now put multiattach in that equation lol.. | |
| 00:35:27 | melwitt | but at least I think now I understand why it was added long ago, if detach didn't used to do a no-op if not attached | |
| 00:36:11 | mnaser | Yeah it’s a bit of old code | |
| 01:51:36 | openstackgerrit | Merged openstack/nova stable/queens: Cleanup the manage-volumes admin doc https://review.openstack.org/545141 | |
| 01:51:46 | openstackgerrit | Merged openstack/nova stable/queens: Add admin guide doc on volume multiattach support https://review.openstack.org/545142 | |
| 02:29:50 | mriedem1 | cfriesen: the method on the instance is just a helper method | |
| 02:30:02 | mriedem | melwitt: mnaser: did you sort out the available volume thing? | |
| 02:30:51 | mriedem | tonyb: done | |
| 02:32:27 | tonyb | mriedem: Thanks. | |
| 02:34:23 | mriedem | imacdonn: if you can, leave a comment on the patch that fixed the issue for you to support it's greatness | |
| 02:34:53 | mriedem | imacdonn: nvm i see you already did, thanks | |
| 02:35:36 | mriedem | lyarwood: can you hit the +2ed stable/queens backports in your morning? https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/queens - then we'll cut RC2 | |
| 02:35:44 | mriedem | tonyb: or i guess you could do that ^ | |
| 02:36:25 | tonyb | mriedem: Oh I can? I didn't think I could between rc1 and release | |
| 02:36:39 | mriedem | smcginnis was able to, so i assume you can | |
| 02:36:51 | tonyb | mriedem: Oh look at that ... | |
| 02:41:22 | mnaser | mriedem: i think so? | |
| 02:45:47 | tonyb | mriedem: done | |
| 02:50:42 | mriedem | tonyb: nice thanks | |
| 02:51:32 | tonyb | mriedem: If you create the tag request with a -W I can update it with the final SHA once they merge ... or we can wait 'til next week | |
| 02:54:58 | openstackgerrit | Merged openstack/osc-placement master: tox.ini settings for global constraints are out of date https://review.openstack.org/543348 | |
| 02:55:30 | mriedem | i shall do so | |
| 02:56:47 | tonyb | mriedem: cool | |
| 02:59:18 | tonyb | mriedem, stephenfin, melwitt: I seem to recall that we said for backports of docs only changes we'd only require 1 stable-core to approve. Did I make that up? | |
| 03:02:07 | mriedem | idk | |
| 03:02:22 | mriedem | if it's a docs bug fix then i think that's probably ok | |