| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-28 | |||
| 14:07:30 | cdent | zigo: i would have flailed | |
| 14:07:37 | zigo | mriedem: I don't think so, it just went into our availability zone called "AZ3" ... | |
| 14:08:02 | zigo | So I get "properties group='AZ3'" when doing server show. | |
| 14:10:36 | mriedem | so it's blowing up here https://github.com/openstack/nova/blob/stable/rocky/nova/objects/request_spec.py#L523 which makes me think the request spec was stored with an instance group without a uuid | |
| 14:10:40 | mriedem | is this an old instance? | |
| 14:11:05 | zigo | mriedem: No. | |
| 14:11:13 | zigo | It's a fairly recent deployment. | |
| 14:11:16 | aarents | Hello! Do you still have any concern regarding this: https://review.opendev.org/#/c/659054 can we workflow+1 ? mriedem | |
| 14:12:32 | mriedem | zigo: so you just created a server in a group and then tried to cold migrate or resize it and it blows up like this? | |
| 14:13:56 | zigo | mriedem: Someone created a server, the compute had a hardware issue, it rebooted, then I get no connectivity with that instance anymore (ovs-vsctl shows some qvo devices missing) so I'm trying to migrate it to fix the networking problem. | |
| 14:14:00 | zigo | Then I get a 500 error ... | |
| 14:14:26 | zigo | I'm trying to cold migrate indeed. | |
| 14:14:48 | zigo | And the trace is what is shown in nova-api.log when doing "nova server migrate" of that instance. | |
| 14:15:09 | mriedem | ok are you able to dump the request_specs entry for that instance? it should be a json blob. | |
| 14:15:25 | zigo | mriedem: How do I do that? | |
| 14:15:31 | zigo | You mean debugging ? | |
| 14:15:45 | zigo | Or just looking in the db? | |
| 14:15:50 | mriedem | select spec from nova_api.request_specs where instance_uuid=<instance id>; | |
| 14:18:40 | zigo | mriedem: http://paste.openstack.org/show/752161/ | |
| 14:19:35 | mriedem | i could see where we could hit an issue by setting an instance_group on the request spec without a uuid here https://github.com/openstack/nova/blob/stable/rocky/nova/objects/request_spec.py#L228 but that is meant to be some legacy code | |
| 14:19:45 | mriedem | which is why i asked if it were an old instance | |
| 14:20:07 | zigo | mriedem: Migrating worked perfectly for other instances ... | |
| 14:20:28 | zigo | mriedem: How do I migrate to another availability zone? | |
| 14:21:15 | mriedem | the server group and the az are different things | |
| 14:21:28 | mriedem | server group is a grouping of ... servers. az is a grouping of compute hosts | |
| 14:21:37 | zigo | Right. | |
| 14:21:38 | zigo | :) | |
| 14:21:49 | mriedem | since this server was created in a specific az, you can't migrate it out of that az unless you bypass the az filter in the scheduler | |
| 14:22:08 | mriedem | like by forcefully bypassing the scheduler, which you can't do with cold migrate | |
| 14:22:45 | zigo | mriedem: I find it annoying that it seems like Nova picks-up an availability zone and stick every instance to it, even if in some cases, I don't want it to do that. | |
| 14:22:48 | mriedem | there is definitely a busted request spec for this instance though http://paste.openstack.org/show/752162/ | |
| 14:23:10 | mriedem | zigo: nova doesn't make servers go into an az by default | |
| 14:23:17 | zigo | Ah... | |
| 14:23:19 | zigo | :) | |
| 14:23:24 | mriedem | the user either requests an az during server create or the operator configures the cloud to default to a specific az | |
| 14:24:03 | mriedem | in fact i have a docs patch for that :) https://review.opendev.org/#/c/650444/2 | |
| 14:25:22 | sean-k-mooney | yonglihe_: comments are on the spec. mainly some typos other then that the content seams fine | |
| 14:25:54 | zigo | Weird, the instance spec shows: | |
| 14:25:54 | zigo | "group": [ "295c99ea-2db6-469a-877f-454a3903a8d8" ] (which has the name "dockerdev") | |
| 14:25:54 | zigo | However, nova server show tells me: group='AZ3' | |
| 14:25:57 | mriedem | zigo: i think somehow we're hitting this https://github.com/openstack/nova/blob/stable/rocky/nova/objects/request_spec.py#L228 and that's causing the problem, but i'm not sure how yet | |
| 14:27:12 | mriedem | i can see AZ3 is the az from the request spec | |
| 14:27:13 | mriedem | "availability_zone": "AZ3", | |
| 14:27:39 | zigo | Yes. | |
| 14:31:16 | mriedem | i'm not sure what that group field is in nova server show... | |
| 14:31:23 | mriedem | nova cli or osc? | |
| 14:32:22 | zigo | osc | |
| 14:32:25 | mriedem | zigo: do you know if this server has been moved before? | |
| 14:32:40 | zigo | mriedem: I don't think it has. | |
| 14:32:54 | zigo | It has not. | |
| 14:33:01 | mriedem | did you check the instance actions records? | |
| 14:34:18 | zigo | mriedem: Nothing except my attempts. | |
| 14:35:23 | mriedem | and you're sure this instance was created from that rocky deployment? | |
| 14:37:19 | mriedem | i think we're somehow saving off a half-baked RequestSpec.instance_group after one failed attempt maybe and then that's causing a new issue loading the group from that busted request spec where the group.uuid isn't set | |
| 14:37:25 | zigo | mriedem: Yes, this cloud deployment has never been using something older. | |
| 14:37:46 | mriedem | i.e. we're hitting https://github.com/openstack/nova/blob/stable/rocky/nova/objects/request_spec.py#L228, saving that, and then failing later here https://github.com/openstack/nova/blob/stable/rocky/nova/objects/request_spec.py#L523 | |
| 14:38:02 | zigo | mriedem: Can I somehow just patch my request_spec db entry? | |
| 14:39:04 | mriedem | you'd have to load http://paste.openstack.org/show/752161/ from json, update the instance_group["nova_object.data"] and set the uuid field to "295c99ea-2db6-469a-877f-454a3903a8d8" | |
| 14:39:07 | mriedem | to match that group scheduler hint | |
| 14:39:13 | mriedem | and then serialize and update that db record | |
| 14:43:17 | mriedem | could also be hitting a reschedule loop which in rocky, the compute didn't pass the request spec back to conductor on reschedule (during resize/cold-migrate) so conductor has to build it's own request spec, that might be why we're only getting a half-baked version | |
| 14:45:38 | mriedem | anyway, i'm not sure what the recreate is, but i can definitely see where we could not have a requestspec.instance_group.uuid so we should probably have a check for that in RequestSpec._from_db_object | |
| 14:45:45 | mriedem | zigo: have you reported a bug? | |
| 14:46:21 | zigo | mriedem: I don't think I'd be a good reporter for this one, I don't know enough about the internals of this request_spec stuff to know what's going on. | |
| 14:46:29 | zigo | That's the first time I see this json output in fact. | |
| 14:47:05 | mriedem | you don't need to know the internals to report the bug, just what is happening and how you think you got here | |
| 14:47:18 | zigo | Ok. | |
| 14:47:31 | zigo | mriedem: Has nova moved to storyboard yet? | |
| 14:47:36 | mriedem | no | |
| 14:47:39 | mriedem | still lp | |
| 14:48:38 | mriedem | my guess is something like this happened: 1. create server in a group, 2. cold migrate the server which fails on host A and does a reschedule to host B which maybe also fails (would be good to know if previous cold migration attempts failed with reschedules), 3. try to cold migrate again which fails with the instance_group.uuid thing | |
| 14:49:30 | mriedem | the reschedule might be the key b/c like i said conductor has to rebuild a request spec and i think that's probably where we're doing a partial build of the request spec but missing the group uuid | |
| 14:56:23 | zigo | mriedem: https://bugs.launchpad.net/nova/+bug/1830747 | |
| 14:56:24 | openstack | Launchpad bug 1830747 in OpenStack Compute (nova) "Error 500 trying to migrate an instance after wrong request_spec" [Undecided,New] | |
| 14:56:26 | mriedem | zigo: yup, thanks | |
| 14:56:35 | zigo | mriedem: I'm trying to add the uuid to the db entry, as you suggested. | |
| 15:03:06 | zigo | mriedem: I think it all starts with this: | |
| 15:03:06 | zigo | 2019-05-28 15:02:35.534 30706 ERROR nova.compute.manager [instance: ae6f8afe-9c64-4aaf-90e8-be8175fee8e4] nova.exception.UnableToMigrateToSelf: Unable to migrate instance (ae6f8afe-9c64-4aaf-90e8-be8175fee8e4) to current host (clint1-compute-5.infomaniak.ch). | |
| 15:03:40 | zigo | (not sure) | |
| 15:04:35 | mriedem | are you requesting a target host when doing the cold migration? | |
| 15:08:18 | zigo | Nop. | |
| 15:08:26 | zigo | There's 3 computes on this AZ. | |
| 15:09:03 | zigo | mriedem: I just migrated another VM on the same compute, and it went fine. | |
| 15:09:50 | zigo | For that one instance, I had request_spec: "instance_group": null | |
| 15:10:00 | mriedem | yeah so that server wasn't in a group | |
| 15:10:12 | zigo | Right. | |
| 15:10:21 | zigo | mriedem: Maybe I can just remove the group from that other instance then? | |
| 15:10:28 | zigo | (that one failing ...) | |
| 15:10:44 | mriedem | that would break the affinity policy for that server | |
| 15:13:01 | mriedem | zigo: i can generate a patch for you quick if you're in a position to try it | |
| 15:13:20 | zigo | I can ... :) | |
| 15:13:28 | mriedem | ok i'll work on something quick | |
| 15:13:43 | zigo | I can always apt-get install --reinstall python3-nova to go back to what it was. | |
| 15:13:50 | zigo | (safely) | |
| 15:14:15 | zigo | mriedem: I'll have to go back home in 40 minutes though. | |
| 15:15:26 | openstackgerrit | Merged openstack/python-novaclient master: Add a description of --on-shared-storage https://review.opendev.org/661527 | |
| 15:16:13 | mriedem | ack, should have this up in 5 min | |
| 15:18:53 | openstackgerrit | Adam Spiers proposed openstack/nova master: extract baselineCPU API call from _get_cpu_traits() https://review.opendev.org/661782 | |
| 15:20:12 | openstackgerrit | Adam Spiers proposed openstack/nova master: extract baselineCPU API call from _get_cpu_traits() https://review.opendev.org/661782 | |
| 15:20:34 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add functional test for bug 1550919 https://review.opendev.org/631294 | |