| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-28 | |||
| 14:47:36 | mriedem | no | |
| 14:47:39 | mriedem | still lp | |
| 14:48:38 | mriedem | my guess is something like this happened: 1. create server in a group, 2. cold migrate the server which fails on host A and does a reschedule to host B which maybe also fails (would be good to know if previous cold migration attempts failed with reschedules), 3. try to cold migrate again which fails with the instance_group.uuid thing | |
| 14:49:30 | mriedem | the reschedule might be the key b/c like i said conductor has to rebuild a request spec and i think that's probably where we're doing a partial build of the request spec but missing the group uuid | |
| 14:56:23 | zigo | mriedem: https://bugs.launchpad.net/nova/+bug/1830747 | |
| 14:56:24 | openstack | Launchpad bug 1830747 in OpenStack Compute (nova) "Error 500 trying to migrate an instance after wrong request_spec" [Undecided,New] | |
| 14:56:26 | mriedem | zigo: yup, thanks | |
| 14:56:35 | zigo | mriedem: I'm trying to add the uuid to the db entry, as you suggested. | |
| 15:03:06 | zigo | 2019-05-28 15:02:35.534 30706 ERROR nova.compute.manager [instance: ae6f8afe-9c64-4aaf-90e8-be8175fee8e4] nova.exception.UnableToMigrateToSelf: Unable to migrate instance (ae6f8afe-9c64-4aaf-90e8-be8175fee8e4) to current host (clint1-compute-5.infomaniak.ch). | |
| 15:03:06 | zigo | mriedem: I think it all starts with this: | |
| 15:03:40 | zigo | (not sure) | |
| 15:04:35 | mriedem | are you requesting a target host when doing the cold migration? | |
| 15:08:18 | zigo | Nop. | |
| 15:08:26 | zigo | There's 3 computes on this AZ. | |
| 15:09:03 | zigo | mriedem: I just migrated another VM on the same compute, and it went fine. | |
| 15:09:50 | zigo | For that one instance, I had request_spec: "instance_group": null | |
| 15:10:00 | mriedem | yeah so that server wasn't in a group | |
| 15:10:12 | zigo | Right. | |
| 15:10:21 | zigo | mriedem: Maybe I can just remove the group from that other instance then? | |
| 15:10:28 | zigo | (that one failing ...) | |
| 15:10:44 | mriedem | that would break the affinity policy for that server | |
| 15:13:01 | mriedem | zigo: i can generate a patch for you quick if you're in a position to try it | |
| 15:13:20 | zigo | I can ... :) | |
| 15:13:28 | mriedem | ok i'll work on something quick | |
| 15:13:43 | zigo | I can always apt-get install --reinstall python3-nova to go back to what it was. | |
| 15:13:50 | zigo | (safely) | |
| 15:14:15 | zigo | mriedem: I'll have to go back home in 40 minutes though. | |
| 15:15:26 | openstackgerrit | Merged openstack/python-novaclient master: Add a description of --on-shared-storage https://review.opendev.org/661527 | |
| 15:16:13 | mriedem | ack, should have this up in 5 min | |
| 15:18:53 | openstackgerrit | Adam Spiers proposed openstack/nova master: extract baselineCPU API call from _get_cpu_traits() https://review.opendev.org/661782 | |
| 15:20:12 | openstackgerrit | Adam Spiers proposed openstack/nova master: extract baselineCPU API call from _get_cpu_traits() https://review.opendev.org/661782 | |
| 15:20:34 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add functional test for bug 1550919 https://review.opendev.org/631294 | |
| 15:20:35 | openstackgerrit | Matthew Booth proposed openstack/nova master: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/578846 | |
| 15:20:35 | openstack | bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9) | |
| 15:22:05 | aspiers | kashyap: hope that refactoring is OK ^^^ | |
| 15:27:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Workaround missing RequestSpec.instance_group.uuid https://review.opendev.org/661786 | |
| 15:27:37 | mriedem | zigo: ^ needs tests and to be prettied up but it should workaround the issue given the request spec you showed me | |
| 15:29:00 | kashyap | aspiers: Hiya | |
| 15:31:20 | zigo | mriedem: Is this for nova-conductor ? | |
| 15:31:26 | zigo | In which node should I apply it? | |
| 15:31:52 | mriedem | api and conductor | |
| 15:32:08 | zigo | mriedem: Ok, then restart conductor and API, I guess ... | |
| 15:32:14 | mriedem | yup | |
| 15:34:18 | kashyap | aspiers: Nice; no test to prove that "nothing is broken" with the refactoring, I presume? | |
| 15:35:02 | aspiers | kashyap: the existing tests should cover it. Refactoring => no change in functionality to test :) | |
| 15:35:49 | kashyap | aspiers: Right | |
| 15:36:59 | kashyap | I like the change, it makes things clearer. (And I intend to refactor _get_guest_baseline_cpu_features() in turn later as part of the CPU modelling spec) | |
| 15:38:54 | aspiers | kashyap: the follow-up patch to add the SEV trait will rename _get_cpu_traits() to _get_cpu_feature_traits() and _get_cpu_traits() will wrap around it | |
| 15:40:42 | kashyap | aspiers: Yep, nod. | |
| 15:40:54 | aspiers | kashyap: coming up in a few secs | |
| 15:41:21 | gibi | mriedem: I've fixed your comments in https://review.opendev.org/#/c/652608/ | |
| 15:41:41 | zigo | mriedem: I still get a 500 error ... | |
| 15:43:48 | mriedem | zigo: do you have a paste of the logs? | |
| 15:43:55 | mriedem | note i added some debug logging | |
| 15:44:01 | zigo | mriedem: name "LOG" is undefined ... | |
| 15:44:09 | zigo | Problem with the logging ... :P | |
| 15:44:24 | openstackgerrit | Adam Spiers proposed openstack/nova master: Provide HW_CPU_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | |
| 15:44:30 | mriedem | ah the LOG didn't exist in rocky... | |
| 15:44:34 | mriedem | i'll remove the logging | |
| 15:44:57 | aspiers | kashyap: https://review.opendev.org/#/c/638680/15/nova/virt/libvirt/driver.py@9638 | |
| 15:45:16 | aspiers | kashyap: keeps the SEV bit nicely separated from the CPU feature code | |
| 15:46:00 | aspiers | efried: this should make you happy too :) | |
| 15:46:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Workaround missing RequestSpec.instance_group.uuid https://review.opendev.org/661786 | |
| 15:46:08 | mriedem | zigo: ^ | |
| 15:46:13 | mriedem | gibi: huh i thought that was approved by now | |
| 15:46:52 | efried | aspiers: Sans click, I'm still gonna say "what about the HW_CPU_X86_AMD_SEV trait?" | |
| 15:47:00 | kashyap | aspiers: A naming nit: "_kernel_supports_amd_sev" --> "_kernel_support_for_amd_sev"? | |
| 15:47:16 | aspiers | efried: haha, thanks for the reminder | |
| 15:47:17 | gibi | mriedem: I would like to get it approved :) | |
| 15:47:25 | aspiers | kashyap: why? | |
| 15:47:31 | efried | aspiers: Maybe kashyap incorporated it in his patch, I haven't looked yet. | |
| 15:48:09 | kashyap | aspiers: Maybe not. It returns either true or false, so as it stands is okay :-) | |
| 15:48:14 | zigo | mriedem: Thanks a lot, unfortunately, I have to go, so I'll try tomorrow. | |
| 15:48:45 | kashyap | efried: Morning; I addressed your remarks and made them part of the older PS; and even added a note in your patch to adjust the Depends-On ID | |
| 15:49:09 | efried | kashyap: ack, saw it go by, haven't looked yet. Will take me a day or two to catch up from "holiday". | |
| 15:49:18 | efried | kashyap: and thank you :) | |
| 15:49:23 | kashyap | efried: Take your sweet time | |
| 15:49:55 | aspiers | efried: OK now here's a reoccurrence of that gerrit limitation we discussed before, but it's worse this time, because I'm not in control of all the reviews involved | |
| 15:50:07 | efried | aspiers: which one? | |
| 15:50:16 | efried | oh, can't branch upside down? | |
| 15:50:19 | aspiers | right | |
| 15:50:26 | mriedem | zigo: ack - also i have triaged how we get here in rocky | |
| 15:50:32 | mriedem | zigo: will have the details in the bug report | |
| 15:50:38 | mriedem | it | |
| 15:50:46 | mriedem | it's the reschedule and such that is the problem as i suspected | |
| 15:50:55 | mriedem | request spec bites me in the ass yet again | |
| 15:50:55 | aspiers | 638680 depends on some of my stuff (e.g. the patch_open helper and the _get_cpu_traits() refactor I just submitted), but now it also needs to depend on kashyap's traits reworking | |
| 15:51:25 | aspiers | efried: which are distinctly orthogonal strands of development | |
| 15:51:46 | efried | aspiers: traits reworking in os-traits or traits reworking in nova? | |
| 15:52:06 | aspiers | efried: ahhh good point! can use Depends-On here | |
| 15:52:15 | efried | aspiers: Well, theoretically yes | |
| 15:52:16 | aspiers | ... I think | |
| 15:52:23 | efried | except I figured out last week that it doesn't bloody work | |
| 15:52:27 | aspiers | :-/ | |
| 15:52:33 | aspiers | I'll give it a go anyway | |
| 15:52:55 | efried | You would have to hack up .zuul.yaml to make os-traits a required project for all the jobs. | |
| 15:53:01 | efried | which is a regal PITA | |
| 15:53:16 | aspiers | why? | |
| 15:53:26 | efried | but go ahead and depends-on it and we'll just get the damn thing merged and released and requirementsed | |
| 15:53:32 | efried | why what? | |