| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-08 | |||
| 16:24:12 | prometheanfire | I'm fine waiting til I can reproduce, I've left stuff in debug | |
| 16:26:37 | sean-k-mooney | stephenfin: jangutter so as i was saying on the placmeent channel ... i adressed my nit in https://review.opendev.org/#/c/666387/2 if ye want to take a look at that | |
| 16:28:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Translatable output strings in heal allocation https://review.opendev.org/668925 | |
| 16:33:25 | sean-k-mooney | aspiers: so you know that domaincap api you added for sev ... its broken | |
| 16:34:45 | sean-k-mooney | we cant default to q35 here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L744-L745 the logic breaks if you have emulators installed that dont support it | |
| 16:35:43 | sean-k-mooney | currently we dont call this code in nova yet because your sev stuff that uses it has not merged yet but i tried to use it for my device model filter stuff and it expolodes beautifully | |
| 16:38:02 | efried | mriedem: I'm going to bump current runways by, what say, 2 days due to last week's holiday? | |
| 16:39:28 | mriedem | shrug | |
| 16:39:35 | mriedem | not sure how much it matters | |
| 16:40:50 | efried | aspiers: SEV series is in merge conflict - any chance of getting that rebased soon? | |
| 16:54:01 | sean-k-mooney | mriedem: so quick question. https://review.opendev.org/#/c/659703/8 change the behavior of force config drive to remove the config drive after first boot | |
| 16:55:15 | sean-k-mooney | to me that seams like a regression as it could lead to data lose in the form of injected files if the user does not copy the injected files form the config_drive or if the operator customerises the config dirve contentce with data scripts that run on each boot | |
| 16:57:17 | sean-k-mooney | if you are using the dynamic vendor data and do not have teh metadta service deployed is this not an issue? | |
| 16:58:36 | sean-k-mooney | the injected files case might be ok if that sets instance.config_drive automaticall but i think https://review.opendev.org/#/c/659703/8/nova/virt/configdrive.py@169 might break some usecauses | |
| 16:59:44 | sean-k-mooney | as such im not sure how safe it is to backport this | |
| 17:02:57 | mriedem | "change the behavior of force config drive to remove the config drive after first boot" - you mean if you boot the server with force_config_drive=True, then change to force_config_drive=False? the bug is actually the opposite - they created a server without a config drive, then changed to force_config_drive=True and after that they can't reboot the servers on the host w/o a config drive since the file doesn't exist | |
| 17:03:35 | mriedem | i said on lyarwood's stein backport, "I'd like to move a bit slowly with this one since it's a very latent issue and is a bit of a behavior change, though justified for the reboot issue. It should also be pretty rare (I don't imagine lots of people are changing the force_config_drive value on their computes once they are deployed)." | |
| 17:03:38 | mriedem | so there it's been sitting | |
| 17:04:08 | sean-k-mooney | mriedem: no what im concerned about is the comment sugges that on a reboot the config drive will be removed | |
| 17:04:12 | mriedem | sean-k-mooney: i'm not really following you on what specific use case you think this regressed | |
| 17:04:15 | sean-k-mooney | e.g. its only present on first boot | |
| 17:04:44 | mriedem | what comment? | |
| 17:05:02 | sean-k-mooney | https://review.opendev.org/#/c/660914/1/nova/virt/configdrive.py | |
| 17:05:50 | mriedem | from the bug, if you go from no config drive to a forced config drive and reboot the vm, it pukes in libvirt | |
| 17:05:53 | mriedem | that's the bug they are fixing | |
| 17:05:54 | sean-k-mooney | if we reboot the vm that was started with a config drive launched_at will not be null right | |
| 17:06:22 | sean-k-mooney | no im talkinbg about if you boot a vme with force_config=ture | |
| 17:06:33 | sean-k-mooney | then you reboot it it should continute to have a config drive | |
| 17:06:48 | sean-k-mooney | but instance.launched_at will not be None on the second boot | |
| 17:06:58 | mriedem | you mean on the reboot | |
| 17:07:07 | sean-k-mooney | yes | |
| 17:07:19 | sean-k-mooney | boot with force it will have a config drive | |
| 17:07:24 | mriedem | so boot with force_config_drive=true, change force_config_drive=false, reboot the vm, the config drive isn't in the vm | |
| 17:07:29 | sean-k-mooney | then if you hard reboot it will go away right | |
| 17:07:45 | sean-k-mooney | no config change | |
| 17:07:55 | sean-k-mooney | if you leave it at force_config_drive=true | |
| 17:07:59 | sean-k-mooney | always | |
| 17:08:03 | sean-k-mooney | boot a vm | |
| 17:08:07 | sean-k-mooney | then hard reboot it | |
| 17:08:17 | sean-k-mooney | it should still have a config drive after the reboot | |
| 17:08:33 | sean-k-mooney | but on the reboot launched_at would be non None | |
| 17:09:02 | sean-k-mooney | so and not instance.launched_at would be false | |
| 17:09:02 | mriedem | yeah i see what you're saying, and instance.config_drive is only set from the API? | |
| 17:09:19 | sean-k-mooney | i think its set in the flavor or image | |
| 17:09:23 | sean-k-mooney | actully no | |
| 17:09:25 | sean-k-mooney | you are right | |
| 17:09:28 | sean-k-mooney | the api | |
| 17:09:43 | sean-k-mooney | so im thinking about deployment where the metadata service is not deployed | |
| 17:10:04 | sean-k-mooney | so you use force_config_drive=true to make cloud init work and for things like vendor data or device role tagging | |
| 17:10:14 | sean-k-mooney | or file injection with the v2.1 api | |
| 17:11:27 | sean-k-mooney | the last point is less relevent beacue that is depercated so i would be ok with that on master but im not sure we should backport that change and it might break people on upgrade that dont deploy the metadata service | |
| 17:11:54 | mriedem | yeah as i said i wasn't totally comfortable with backporting it | |
| 17:12:21 | mriedem | oh wait | |
| 17:12:25 | mriedem | i remember digging into this | |
| 17:12:46 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1758 | |
| 17:13:10 | mriedem | instance.config_drive will be set on compute if force_config_drive=True and the user doesn't specify it in the api | |
| 17:13:30 | mriedem | so i don't think your scenario holds | |
| 17:13:44 | mriedem | https://review.opendev.org/#/c/659703/4/nova/virt/configdrive.py@169 | |
| 17:14:32 | mriedem | so tl;dr, you likely need to recreate whatever regression you think there is and report a bug if we're going to talk about reverting that change | |
| 17:14:40 | mriedem | and i've got a hungry kid here and need to make lunch | |
| 17:15:16 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/configdrive.py#L181 | |
| 17:15:35 | sean-k-mooney | so if we have already set instace.launched_at then it wont | |
| 17:15:58 | sean-k-mooney | mriedem: sure take care of lunch :) | |
| 17:16:34 | mriedem | on first create, if the host has force_config_drive=True, we'll set instance.config_drive=True, and on subsequent calls to update_instance we won't update it because "if not True" is False | |
| 17:16:57 | mriedem | anyway, like i said, this would be easier if you can actually recreate a problem and report a bug rather than both of us mostly just guessing based on code | |
| 17:17:03 | mriedem | but i did look into this earlier on the patch | |
| 17:17:04 | sean-k-mooney | i think we need to reorder 1757 and 1758 https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1758 | |
| 17:17:19 | sean-k-mooney | mriedem: ok no worries | |
| 17:17:33 | sean-k-mooney | i can test this in my devstack setup and see if it causes an issue or not | |
| 17:18:17 | sean-k-mooney | go make food :) ill let you know if i find anything | |
| 17:21:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.opendev.org/645520 | |
| 17:21:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update AZ admin doc to mention the new way to specify hosts https://review.opendev.org/666767 | |
| 17:35:58 | openstackgerrit | Merged openstack/nova stable/rocky: Ignore hw_vif_type for direct, direct-physical vNIC types https://review.opendev.org/667355 | |
| 17:40:24 | sean-k-mooney | ya so with master if you set force_config_drive=true, then boot a new vm and hard reboot it it will not have the config dirve on teh second boot | |
| 17:40:31 | sean-k-mooney | so that intoduced a new bug | |
| 17:41:02 | sean-k-mooney | which should be fixable by swapping instance.launched_at and configdrive.update_instace here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1758 | |
| 17:41:12 | sean-k-mooney | which ill go test now | |
| 17:45:57 | sean-k-mooney | yep that fixes it | |
| 17:46:06 | sean-k-mooney | ill file a bug and push a patch | |
| 17:47:05 | sean-k-mooney | the order didnt matter befoce because required_by did not depend on the launched_at field not it does so we need to set that after | |
| 17:47:24 | sean-k-mooney | ill try these change teh config to diable force and see if ti does the right thing now | |
| 17:54:25 | sean-k-mooney | yep it works corectly after config updte too. old vms keep the config drive an new ones dont have them. and if i go back from false to true old vms dont get a config drive but new vms do | |
| 18:30:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.opendev.org/645520 | |
| 18:30:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update AZ admin doc to mention the new way to specify hosts https://review.opendev.org/666767 | |
| 18:32:47 | spatel | sean-k-mooney: one of my vm stuck in powering-on stat so i did | |
| 18:32:48 | spatel | nova reset-state --active 368f2b45-c4b8-460e-9269-023ef80a69d1 | |
| 18:33:18 | spatel | now now when i start vm it is saying - Cannot 'start' instance 368f2b45-c4b8-460e-9269-023ef80a69d1 while it is in vm_state active (HTTP 409) (Request-ID: req-29b635ac-4d8f-4139-9170-19824f019806) | |
| 18:33:39 | mriedem | i'm +2 on "Add host and hypervisor_hostname flag to create server https://review.opendev.org/645520" now | |
| 18:33:50 | sean-k-mooney | spatel: you will need to hard reboot it | |
| 18:36:28 | spatel | what is the option for command line to hard-reboot? | |
| 18:36:39 | spatel | let me try from GUI | |
| 18:37:12 | sean-k-mooney | "openstack server reboot --hard" but its available in horizon too | |
| 18:42:15 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: make config drives sticky bug 1835822 https://review.opendev.org/669738 | |
| 18:42:16 | openstack | bug 1835822 in OpenStack Compute (nova) "vms loose acess to config drive with CONF.force_config_drive=True after hard reboot" [Medium,Confirmed] https://launchpad.net/bugs/1835822 - Assigned to sean mooney (sean-k-mooney) | |
| 18:42:28 | sean-k-mooney | mriedem: ^ that is the fix for the config drive issue | |
| 18:43:26 | spatel | sean-k-mooney: that works :)_ | |
| 18:43:34 | spatel | you are awesome | |
| 18:43:55 | sean-k-mooney | no i just have hit the same proablem | |
| 18:44:12 | sean-k-mooney | if you use reset-state to set it to active you need to use hard-reboot to fix it | |
| 19:26:04 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add host and hypervisor_hostname to create servers https://review.opendev.org/647671 | |