| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-31 | |||
| 15:36:03 | mriedem | but what gets us back to getting a real vif type from neutron? | |
| 15:36:05 | dansmith | but presumably it was bound before and just restoring the info makes it work on the original host again, | |
| 15:36:06 | dansmith | so presumably if we re-bind on the source it will work | |
| 15:36:07 | mriedem | re-plugging on the source? | |
| 15:36:17 | dansmith | mriedem: well, restarting the compute node for sure :) | |
| 15:36:26 | dansmith | just not positive about the reboot case | |
| 15:36:32 | dansmith | although I think it probably will | |
| 15:36:51 | mriedem | re-plug fails on the source when restarting the compute | |
| 15:36:58 | mriedem | that's what https://review.openstack.org/#/c/587498/1/nova/compute/manager.py is for | |
| 15:37:13 | mriedem | because it's trying to plug using a binding_failed vif type | |
| 15:37:18 | sean-k-mooney | mriedem: repluging is not the same as rebinding however. | |
| 15:37:38 | sean-k-mooney | its using the binidng_failed vif form the info cache to replug right? | |
| 15:37:46 | mriedem | on compute restart yes | |
| 15:37:52 | mriedem | which causes compute to blow up and fail to start | |
| 15:38:19 | sean-k-mooney | ya so we could have it try to rebind the port when that happens for the current host | |
| 15:38:48 | dansmith | oh right, | |
| 15:38:52 | mriedem | i'm failing to see what ever changed the port binding on the dest | |
| 15:38:57 | dansmith | it'll re-plug on restart but likely not bind | |
| 15:39:08 | dansmith | mriedem: the live migration, no? | |
| 15:39:24 | mriedem | pre_live_migration doesn't change the port's host binding before it tries to plug | |
| 15:39:38 | mriedem | the port binding only changes to the dest in post live migration | |
| 15:40:33 | mriedem | he's failing here https://github.com/openstack/nova/blob/f6ce98749a9891a9261e392cde648f55f3dde3fa/nova/compute/manager.py#L6163 | |
| 15:41:04 | mriedem | that's an rpc call from the source host and then the source catches that here https://github.com/openstack/nova/blob/f6ce98749a9891a9261e392cde648f55f3dde3fa/nova/compute/manager.py#L6327 and triggers rollback | |
| 15:41:34 | mriedem | rollback calls this https://github.com/openstack/nova/blob/f6ce98749a9891a9261e392cde648f55f3dde3fa/nova/compute/manager.py#L6859 but it doesn't do anything for neutron | |
| 15:43:13 | mriedem | starting in rocky, https://github.com/openstack/nova/blob/f6ce98749a9891a9261e392cde648f55f3dde3fa/nova/compute/manager.py#L6903 will clear the port's migrating_to profile entry and delete any inactive dest host port bindings | |
| 15:43:36 | dansmith | mriedem: any chance this is the second migration attempt after the first failed in post? | |
| 15:43:50 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Cleanup comp_node, res_prov, services, aggregate_hosts during cell deletion https://review.openstack.org/546660 | |
| 15:44:00 | mriedem | dansmith: possibly | |
| 15:44:13 | mriedem | the info cache could have already had binding_failed in it before they started the live migration | |
| 15:44:17 | mriedem | which makes pre_live_migration fail | |
| 15:44:29 | dansmith | right | |
| 15:45:15 | sean-k-mooney | hum that sound plausible | |
| 15:45:40 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Cleanup comp_node, res_prov, services, aggregate_hosts during cell deletion https://review.openstack.org/546660 | |
| 15:45:50 | sean-k-mooney | is the fix in that case still to just not update the info_cache woith binding_vailed | |
| 15:46:08 | mriedem | idk | |
| 15:46:10 | sean-k-mooney | *binding_failed | |
| 15:46:19 | mriedem | kind of throwing things at the wall at that point trying to see what works | |
| 15:46:23 | mriedem | w/o a real recreate i can tinker with | |
| 15:46:46 | sean-k-mooney | i guess we need to create a regression test for it to figure it out | |
| 15:46:53 | sean-k-mooney | or get logs | |
| 15:47:11 | mriedem | we can't really test this very easily... | |
| 15:47:14 | mriedem | tempest won't do this | |
| 15:47:25 | mriedem | and an in-tree regressoin test would just stub out neutron | |
| 15:48:00 | mriedem | this is why for now i just did handle the error like the original bug for the exact same thing | |
| 15:48:10 | mriedem | https://review.openstack.org/#/c/129158/ | |
| 15:48:18 | mriedem | exact same issue, just pre-os-vif | |
| 15:49:52 | sean-k-mooney | well catching the error and preventing the agent from crashing is proably correct in any case. | |
| 15:56:11 | cdent | is there a tag for nova-api | |
| 15:56:49 | cdent | or does one just say "api"? | |
| 15:57:55 | sean-k-mooney | cdent: i had assumed [openstack-dev][nova][api] ... | |
| 15:58:16 | sean-k-mooney | im probably wrong however | |
| 15:58:24 | mriedem | i think he's talking about bug tags | |
| 15:58:37 | sean-k-mooney | oh ok | |
| 15:58:41 | cdent | sean-k-mooney: yeah, sorry, mriedem is right, I wasn't context switching well | |
| 15:58:50 | cdent | too many contexts at once | |
| 15:59:07 | sean-k-mooney | cdent: then yes its just api https://wiki.openstack.org/wiki/Nova/BugTriage | |
| 15:59:15 | cdent | thanks | |
| 16:01:22 | openstackgerrit | Jay Pipes proposed openstack/nova master: track down why inventory placement calls multiply https://review.openstack.org/587510 | |
| 16:02:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: track down why inventory placement calls multiply https://review.openstack.org/587510 | |
| 16:03:59 | tbachman | !pspy | |
| 16:04:00 | openstack | tbachman: Error: "pspy" is not a valid command. | |
| 16:05:03 | tbachman | !pyspy | |
| 16:05:05 | openstack | tbachman: Error: "pyspy" is not a valid command. | |
| 16:05:54 | dansmith | tbachman: `less ~/.irclogs/...` | |
| 16:06:14 | tbachman | dansmith: thx! | |
| 16:06:22 | dansmith | :P | |
| 16:09:26 | tbachman | there is a command tho — can’t remember what it is/was | |
| 16:10:08 | tbachman | !? | |
| 16:10:09 | openstack | tbachman: Error: "?" is not a valid command. | |
| 16:10:11 | tbachman | heh | |
| 16:10:19 | tbachman | !help | |
| 16:10:20 | openstack | tbachman: (help [<plugin>] [<command>]) -- This command gives a useful description of what <command> does. <plugin> is only necessary if the command is in more than one plugin. | |
| 16:12:16 | sean-k-mooney | tbachman: generally i jsut install highmon in my client that said weechat highlights mentions by default even without it now | |
| 16:12:42 | tbachman | sean-k-mooney: thx! | |
| 16:15:59 | jroll | tbachman: https://anticdent.org/purple-irc-bot.html | |
| 16:16:20 | tbachman | p!spy | |
| 16:16:24 | tbachman | jroll: thx! | |
| 16:16:32 | tbachman | that was the one I was looking for :) | |
| 16:16:36 | jroll | :) | |
| 16:16:49 | cdent | dansmith, mriedem: rado filed a bug on that threading/eventlet thing: https://bugs.launchpad.net/nova/+bug/1784666 | |
| 16:16:49 | openstack | Launchpad bug 1784666 in OpenStack Compute (nova) "The thread local which stores the request context is not green" [Medium,New] | |
| 16:16:50 | melwitt | cdent, mriedem: I have a newer change I'm experimenting with for the request_id's jumping across requests. some have reported that it's not only restricted to periodics and that it's happening in general with parallel requests. the patch I'm testing is https://review.openstack.org/582332 but it's not clear to me how to verify whether it fully solves the problem | |
| 16:17:17 | cdent | wow, major jinx with melwitt there | |
| 16:17:42 | dansmith | melwitt: it happens in compute, which never uses target_cell, so I wouldn't think that is related | |
| 16:18:02 | cdent | melwitt: rado is pretty sure it is "simply" an import ordering problem | |
| 16:18:22 | cdent | he can fix it by changing things around | |
| 16:18:23 | dansmith | like the comment on that patch says | |
| 16:18:24 | melwitt | oh, yeah okay. that makes more sense | |
| 16:19:03 | melwitt | ah, I missed that comment | |
| 16:19:28 | mriedem | dansmith: melwitt: cdent: yeah says the same as radu's bug | |
| 16:19:39 | mriedem | "In his patch, eventlet.monkey_patch() must before import oslo_log (actually before import oslo_context)." | |
| 16:19:43 | dansmith | yeah | |
| 16:22:28 | melwitt | unrelated to that, I'd think the cell targeting would cause request_id changes in logging in nova-api though. but I'm not sure I've observed that before. I've seen only the ones in compute | |
| 16:23:26 | dansmith | melwitt: but any targeting of a context should result in the same request id right? | |
| 16:23:45 | dansmith | so even if you did overwrite the logging context with the targeted one, it shouldn't matter I would think | |
| 16:24:23 | melwitt | dansmith: yeah, maybe that's why I haven't observed it | |
| 16:24:27 | dansmith | yeah :) | |
| 16:24:46 | melwitt | mriedem: do we want to dupe one of these to the other bug? https://bugs.launchpad.net/nova/+bug/1773102 | |
| 16:24:46 | openstack | Launchpad bug 1773102 in OpenStack Compute (nova) "Abnormal request id in logs" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 16:28:49 | mriedem | melwitt: yes done | |