| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-19 | |||
| 17:32:13 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Re-propose nested resource providers spec https://review.openstack.org/505209 | |
| 17:32:45 | edleafe | cfriesen: I know that when I worked with RAX regions, they were | |
| 17:55:35 | tasker | the live-migration of the instance that was interrupted during the "post-migrtaion" tasks and the BINDING_PROFILE=None exception, the instance moved to the target host, but nova was interrupted and didn't update the database. nova thinks the instance is still on the original host. how can I get nova to update itself? | |
| 17:56:16 | tasker | restarting the `nova-compute` service did not do what I thought it might -- look to see what instances were active on the compute-host and update the tables accordingly. | |
| 17:58:09 | tasker | restarting results in "While synchronizing instance power states, found 2 instances in the database and 1 instances on the hypervisor." followed by "Instance is unexpectedly not found. Ignore." | |
| 17:58:43 | tasker | that's on the original host. the destination host has "While synchronizing instance power states, found 3 instances in the database and 4 instances on the hypervisor." and nothing else. | |
| 18:02:05 | tasker | i found https://bugs.launchpad.net/nova/+bug/1288958 on the same "Instance is unexpectedly not found. Igonre." message, but that one expired in April `16. the steps disclosed there are different but the state of Nova is similar. | |
| 18:02:10 | openstack | Launchpad bug 1288958 in OpenStack Compute (nova) "nova out of sync with the hypervisor " [Medium,Expired] | |
| 18:02:52 | tasker | nova list shows my particular instance with "power state = NOSTATE" | |
| 18:12:35 | melwitt | tasker: was it the 'setup_networks_on_host' that failed during post_live_migration_at_destination? because if so, that means the network failed setup on the destination and probably isn't working properly on the target | |
| 18:13:32 | melwitt | (I would think, I'm not a live migration expert) | |
| 18:20:11 | tasker | I don't see `setup_networks_on_host` in the traceback. it's "post_live_migration_at_destination -> migrate_instance_finish -> _update_port_binding_for_instance" | |
| 18:20:29 | tasker | do you have a pastie site that you prefer to use? I can flip you the traceback. | |
| 18:22:19 | melwitt | okay, I see the network_api.migrate_instance_finish call (just looking at the code). it's part of the network setup AFAICT | |
| 18:23:11 | melwitt | we use paste.openstack.org and pastebin.com | |
| 18:24:20 | melwitt | but just looking at the code, you're seeing it fail during the network setup at the destination, which would mean it's not going to work properly there, so it's not updating it to the target | |
| 18:25:57 | melwitt | yeah, the migrate_instance_finish is what updates the port binding for the instance (see nova/network/neutronv2/api.py) | |
| 18:27:46 | melwitt | if that failed, I don't think it would be correct to update the instance host to the target host. so AFAICT, the instance is still on the original host and there's a half-baked instance on the destination which needs to be cleaned up | |
| 18:28:16 | tasker | virsh list shows no instance on the original host. | |
| 18:28:49 | tasker | it does show it at the target host, but I cannot tell what state it's in. | |
| 18:28:52 | melwitt | *looking at more code* yeah, I was trying to see what has happened to the libvirt domain by this point | |
| 18:31:33 | tasker | melwitt: http://paste.openstack.org/show/621469/ | |
| 18:31:38 | melwitt | so it moved the domain, failed to update the port binding, so it's not in a working state. so far I'm not seeing how you could recover from that | |
| 18:32:13 | tasker | deletion is recovery, right? <G> | |
| 18:33:45 | melwitt | heh | |
| 18:34:06 | openstackgerrit | Claudiu Belu proposed openstack/nova master: hyperv: report disk_available_least field https://review.openstack.org/504904 | |
| 18:38:03 | tasker | this is a test cluster so no real worries. | |
| 18:42:00 | melwitt | okay, well that's good. the live migration flow is known to be gnarly and if anything fails in the middle of it, manual intervention is needed to fix things. I'm not familiar with more detail about it | |
| 18:42:35 | melwitt | from the trace you posted though, it looks like there's a bug in assuming binding_profile can't be None | |
| 18:43:18 | melwitt | er, actually it should be able to be assumed because: binding_profile = p.get(BINDING_PROFILE, {}) | |
| 18:45:26 | tasker | yeah, mriedem and I have addressed that with https://review.openstack.org/#/c/504260 | |
| 18:46:19 | melwitt | oh, I see. it's set but it's set to None. got it | |
| 18:46:46 | tasker | thanks for your help melwitt. I'm going to work on getting rid of this now broken instance. | |
| 18:53:36 | openstackgerrit | Merged openstack/nova stable/pike: Target context when setting instance to ERROR when over quota https://review.openstack.org/504178 | |
| 19:03:15 | sdague | mriedem: https://review.openstack.org/#/c/454323/ that's the defaulting live snapshot to true | |
| 19:15:05 | tasker | I've got one instance left that's not migrating. the migration request is accepted by the API server, but after that it just disappears. `nova migration-list` immediately show is as error but I see no errors logged in the api containers and the message doesn't get to the compute hosts. I'm stumped on this one. I have no tracebacks or logs to show. | |
| 19:15:22 | tasker | messages disappearing are usually related to rabbit communications. | |
| 19:15:32 | tasker | but all other migrations worked just fine. except this instance. | |
| 19:16:14 | tasker | this instance was made when the cluster was at mitaka / trusty. | |
| 19:41:16 | melwitt | tasker: I think next it goes to conductor so check your nova-conductor logs for errors | |
| 19:41:34 | tasker | I have not. I will check now. | |
| 19:42:45 | tasker | you are correct. looking deeper. | |
| 19:43:06 | tasker | g'ah. "no valid host was found". | |
| 19:43:49 | tasker | same, regardless of which host I try and send it to. ( i only have 3 hypervisors ). | |
| 19:45:01 | melwitt | okay, to dig further, you need to look at nova-scheduler debug logs. if you don't already have log level DEBUG enabled, you'll need to configure that and restart nova-scheduler | |
| 19:45:14 | melwitt | there you would be able to see which scheduler filters failed and why | |
| 19:49:47 | openstackgerrit | Dan Smith proposed openstack/nova master: Add base implementation for efficient cross-cell instance listing https://review.openstack.org/504983 | |
| 19:49:47 | openstackgerrit | Dan Smith proposed openstack/nova master: Make instance_list honor global query limit https://review.openstack.org/504984 | |
| 19:49:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Add db.instance_get_by_sort_filters() https://review.openstack.org/504985 | |
| 19:49:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Support pagination in instance_list https://review.openstack.org/504986 | |
| 19:49:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Add fault-filling into instance_get_all_by_filters_sort() https://review.openstack.org/505391 | |
| 19:49:49 | openstackgerrit | Dan Smith proposed openstack/nova master: Add tests to validate instance_list handles faults correctly https://review.openstack.org/505392 | |
| 19:57:11 | sdague | efried: looks like you have to fix some unit tests on the remove | |
| 19:57:29 | efried | sdague Sorry, which change set are we talking about? | |
| 19:57:47 | efried | oh, https://review.openstack.org/#/c/505317/ ? | |
| 19:57:56 | sdague | yeh | |
| 19:58:21 | efried | bah. I took a quick look for stuff I thought might be affected, but didn't run full tox locally. | |
| 19:58:28 | efried | You know, cause of the heat footprint :) | |
| 19:58:32 | efried | will fix. | |
| 19:58:34 | sdague | heh | |
| 20:15:45 | dansmith | gdi, pep8.. good for nothing. | |
| 21:01:19 | openstackgerrit | Eric Fried proposed openstack/nova master: Don't fix protocol-less glance api_servers anymore https://review.openstack.org/505317 | |
| 21:01:23 | mriedem | is it just me or is the live migration job failing a lot now | |
| 21:01:40 | efried | sdague -^ fixed, I believe. | |
| 21:09:02 | mikal | mriedem: yeah, I should send that heads up email as discussed. I will do that tonight / tomorrow. | |
| 21:09:17 | mikal | In other news, is there a way to search for patches containing certain text? | |
| 21:09:28 | mikal | I want to find the couple of patches that renaming the privsep contexts broke | |
| 21:09:33 | mriedem | mikal: i did put something about privsep in my last recap email | |
| 21:09:45 | mriedem | mikal: search in the commit message? | |
| 21:10:05 | mikal | mriedem: yeah, I saw your para. I'll do something more explicit as well. | |
| 21:10:22 | mikal | mriedem: it might not be in the commit message though. I suspect I get to download every outstanding ref and grep them. | |
| 21:12:10 | mriedem | ask clarkb | |
| 21:12:50 | clarkb | mikal: mriedem https://review.openstack.org/Documentation/user-search.html | |
| 21:13:17 | openstackgerrit | Dan Smith proposed openstack/nova master: Add base implementation for efficient cross-cell instance listing https://review.openstack.org/504983 | |
| 21:13:17 | openstackgerrit | Dan Smith proposed openstack/nova master: Make instance_list honor global query limit https://review.openstack.org/504984 | |
| 21:13:18 | openstackgerrit | Dan Smith proposed openstack/nova master: Add db.instance_get_by_sort_filters() https://review.openstack.org/504985 | |
| 21:13:18 | openstackgerrit | Dan Smith proposed openstack/nova master: Support pagination in instance_list https://review.openstack.org/504986 | |
| 21:13:19 | openstackgerrit | Dan Smith proposed openstack/nova master: Add fault-filling into instance_get_all_by_filters_sort() https://review.openstack.org/505391 | |
| 21:13:19 | openstackgerrit | Dan Smith proposed openstack/nova master: Add tests to validate instance_list handles faults correctly https://review.openstack.org/505392 | |
| 21:13:56 | mikal | clarkb: I don't see a "contains text" or equivalent there. Am I going blind? | |
| 21:14:49 | clarkb | I'm not sure | |
| 21:14:57 | clarkb | we just upgraded and I haven't had a chance t oread the new search stuff | |
| 21:15:58 | clarkb | looks like you can do path, filename, commit message, and diff lines, but not diff line content | |
| 21:16:27 | mikal | clarkb: yeah, so that doesn't save me | |
| 21:16:39 | clarkb | codesearch.openstack.org may be of use | |
| 21:16:41 | mikal | clarkb: I might script downloading every patch as a diff and then just grep | |
| 21:16:51 | mikal | Does codesearch do unmerged patches? | |
| 21:16:57 | clarkb | oh they are unmerged, no it won't | |
| 21:17:30 | mikal | :( | |
| 21:22:55 | openstackgerrit | Jackie Truong proposed openstack/python-novaclient master: Microversion 2.54 - Add trusted_certificates param https://review.openstack.org/500396 | |
| 21:33:09 | mriedem | efried: a few questions in https://review.openstack.org/#/c/500190/ but +2 lgtm | |
| 21:33:27 | efried | mriedem Roger wilco, and thanks. | |
| 21:47:20 | openstackgerrit | Dan Smith proposed openstack/nova master: Add get_instance_objects_sorted() https://review.openstack.org/505417 | |
| 21:47:20 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Use improved instance_list module in compute API https://review.openstack.org/505418 | |
| 22:09:46 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Spec: Use keystoneauth1 Adapter for endpoints https://review.openstack.org/500190 | |
| 22:40:20 | dansmith | mriedem: is the live migration job known to have issues? | |
| 22:40:32 | dansmith | it's failing on several patches that aren't changing anything | |
| 23:48:49 | alex_xu | mriedem: thanks, I will check then today | |
| 23:51:49 | catintheroof | i guys, quick question, i have nova-api sitting behind haproxy doing SSL termination, and the nova-api keystone endpoint is https://, why when i call https://[HAPROXY_VIP]:8774 it responses on the body that nova is listening on http ? | |
| 23:51:59 | catintheroof | any hint or help ? | |