Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-07-08
16:11:25 prometheanfire mriedem: no, volume created manually (worked), then nova was asked to attach volume (failed)
16:11:32 prometheanfire api can't remove it from the instance
16:11:41 prometheanfire the failed part half attached it to the instance
16:12:20 prometheanfire added to block_device_mapping table but no action taken on compute host
16:12:23 openstackgerrit Merged openstack/os-vif master: Add Python 3 Train unit tests https://review.opendev.org/669438
16:12:50 mriedem so there is a bdm, and the volume shows up for the server but when you try to detach it what happens? nova gets an error from cinder?
16:12:58 mriedem saying it's already detached (not attached) or something?
16:13:15 prometheanfire nova questions cinder if it's attached, cinder says no, nova does nothing
16:14:00 mriedem as in nova doesn't delete the bdm so it stops showing up as attached to the server
16:14:03 prometheanfire does nothing -> says it's not attached, why are you asking me to detach something that's not attached (returned to user)
16:14:15 mriedem which release?
16:14:19 prometheanfire stein
16:14:50 prometheanfire I haven't been able to reproduce this after restarting nova stuff infra side
16:15:01 prometheanfire so maybe some timeout or hanging connection?
16:15:20 mriedem my guess is cinder is failing on this call: https://github.com/openstack/nova/blob/86524773b8cd3a52c98409c7ca183b4e1873e2b8/nova/compute/api.py#L4175
16:15:43 prometheanfire maybe, for the detach part
16:16:14 mriedem well what error do you get from the compute API?
16:16:26 mriedem "Invalid volume: %(reason)s"
16:16:53 prometheanfire No volume with a name or ID of '3e92f9b6-1a3b-4a7e-8487-6ff253e888db' exists.
16:17:01 prometheanfire since I removed the volumes
16:17:36 prometheanfire Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-01d8a3c5-66f4-43a4-bac5-9f1104a292fe) (HTTP 400) (Request-ID: req-358132e4-38a1-493a-b87b-d135791bae2d)
16:17:40 mriedem msg = _("Unable to detach volume. Volume status must be 'in-use' "
16:17:41 mriedem yup
16:17:41 mriedem "and attach_status must be 'attached' to detach.")
16:17:48 mriedem that's the begin_detaching call failing
16:18:05 prometheanfire because it never actually attached :D
16:18:11 openstackgerrit sean mooney proposed openstack/os-vif master: Sync Sphinx requirement https://review.opendev.org/666387
16:18:14 mriedem right i get it
16:18:41 openstackgerrit Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.opendev.org/637955
16:18:48 gibi mriedem: rebased ^^
16:19:43 mriedem prometheanfire: interesting you wouldn't hit this in the compute service when the attach fails and delete the bdm https://github.com/openstack/nova/blob/86524773b8cd3a52c98409c7ca183b4e1873e2b8/nova/compute/manager.py#L5722
16:20:41 prometheanfire haven't been able to reproduce after restart of services, but am leaving it in debug mode in case it happens again
16:21:06 openstackgerrit Balazs Gibizer proposed openstack/nova master: Move consts from neutronv2/api to constants module https://review.opendev.org/668945
16:21:07 mriedem were there errors in the compute log after the failed attach about not being able to delete the bdm?
16:21:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Use neutron contants in cmd/manage.py https://review.opendev.org/668946
16:21:26 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add 'resource_request' to neutronv2/constants https://review.opendev.org/668947
16:22:04 prometheanfire compute didn't even attempt to attach from what I could see
16:22:12 prometheanfire didn't get that far
16:22:34 mriedem so...the rpc cast from nova api to nova compute failed?
16:22:39 mriedem you said you got a 504 somewhere
16:22:59 prometheanfire 504 from nova-api to client
16:23:05 prometheanfire openstackclient
16:23:34 mriedem ok i don't know why that would happen
16:24:12 prometheanfire I'm fine waiting til I can reproduce, I've left stuff in debug
16:26:37 sean-k-mooney stephenfin: jangutter so as i was saying on the placmeent channel ... i adressed my nit in https://review.opendev.org/#/c/666387/2 if ye want to take a look at that
16:28:08 openstackgerrit Balazs Gibizer proposed openstack/nova master: Translatable output strings in heal allocation https://review.opendev.org/668925
16:33:25 sean-k-mooney aspiers: so you know that domaincap api you added for sev ... its broken
16:34:45 sean-k-mooney we cant default to q35 here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L744-L745 the logic breaks if you have emulators installed that dont support it
16:35:43 sean-k-mooney currently we dont call this code in nova yet because your sev stuff that uses it has not merged yet but i tried to use it for my device model filter stuff and it expolodes beautifully
16:38:02 efried mriedem: I'm going to bump current runways by, what say, 2 days due to last week's holiday?
16:39:28 mriedem shrug
16:39:35 mriedem not sure how much it matters
16:40:50 efried aspiers: SEV series is in merge conflict - any chance of getting that rebased soon?
16:54:01 sean-k-mooney mriedem: so quick question. https://review.opendev.org/#/c/659703/8 change the behavior of force config drive to remove the config drive after first boot
16:55:15 sean-k-mooney to me that seams like a regression as it could lead to data lose in the form of injected files if the user does not copy the injected files form the config_drive or if the operator customerises the config dirve contentce with data scripts that run on each boot
16:57:17 sean-k-mooney if you are using the dynamic vendor data and do not have teh metadta service deployed is this not an issue?
16:58:36 sean-k-mooney the injected files case might be ok if that sets instance.config_drive automaticall but i think https://review.opendev.org/#/c/659703/8/nova/virt/configdrive.py@169 might break some usecauses
16:59:44 sean-k-mooney as such im not sure how safe it is to backport this
17:02:57 mriedem "change the behavior of force config drive to remove the config drive after first boot" - you mean if you boot the server with force_config_drive=True, then change to force_config_drive=False? the bug is actually the opposite - they created a server without a config drive, then changed to force_config_drive=True and after that they can't reboot the servers on the host w/o a config drive since the file doesn't exist
17:03:35 mriedem i said on lyarwood's stein backport, "I'd like to move a bit slowly with this one since it's a very latent issue and is a bit of a behavior change, though justified for the reboot issue. It should also be pretty rare (I don't imagine lots of people are changing the force_config_drive value on their computes once they are deployed)."
17:03:38 mriedem so there it's been sitting
17:04:08 sean-k-mooney mriedem: no what im concerned about is the comment sugges that on a reboot the config drive will be removed
17:04:12 mriedem sean-k-mooney: i'm not really following you on what specific use case you think this regressed
17:04:15 sean-k-mooney e.g. its only present on first boot
17:04:44 mriedem what comment?
17:05:02 sean-k-mooney https://review.opendev.org/#/c/660914/1/nova/virt/configdrive.py
17:05:50 mriedem from the bug, if you go from no config drive to a forced config drive and reboot the vm, it pukes in libvirt
17:05:53 mriedem that's the bug they are fixing
17:05:54 sean-k-mooney if we reboot the vm that was started with a config drive launched_at will not be null right
17:06:22 sean-k-mooney no im talkinbg about if you boot a vme with force_config=ture
17:06:33 sean-k-mooney then you reboot it it should continute to have a config drive
17:06:48 sean-k-mooney but instance.launched_at will not be None on the second boot
17:06:58 mriedem you mean on the reboot
17:07:07 sean-k-mooney yes
17:07:19 sean-k-mooney boot with force it will have a config drive
17:07:24 mriedem so boot with force_config_drive=true, change force_config_drive=false, reboot the vm, the config drive isn't in the vm
17:07:29 sean-k-mooney then if you hard reboot it will go away right
17:07:45 sean-k-mooney no config change
17:07:55 sean-k-mooney if you leave it at force_config_drive=true
17:07:59 sean-k-mooney always
17:08:03 sean-k-mooney boot a vm
17:08:07 sean-k-mooney then hard reboot it
17:08:17 sean-k-mooney it should still have a config drive after the reboot
17:08:33 sean-k-mooney but on the reboot launched_at would be non None
17:09:02 mriedem yeah i see what you're saying, and instance.config_drive is only set from the API?
17:09:02 sean-k-mooney so and not instance.launched_at would be false
17:09:19 sean-k-mooney i think its set in the flavor or image
17:09:23 sean-k-mooney actully no
17:09:25 sean-k-mooney you are right
17:09:28 sean-k-mooney the api
17:09:43 sean-k-mooney so im thinking about deployment where the metadata service is not deployed
17:10:04 sean-k-mooney so you use force_config_drive=true to make cloud init work and for things like vendor data or device role tagging
17:10:14 sean-k-mooney or file injection with the v2.1 api
17:11:27 sean-k-mooney the last point is less relevent beacue that is depercated so i would be ok with that on master but im not sure we should backport that change and it might break people on upgrade that dont deploy the metadata service
17:11:54 mriedem yeah as i said i wasn't totally comfortable with backporting it
17:12:21 mriedem oh wait
17:12:25 mriedem i remember digging into this
17:12:46 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1758
17:13:10 mriedem instance.config_drive will be set on compute if force_config_drive=True and the user doesn't specify it in the api
17:13:30 mriedem so i don't think your scenario holds

Earlier   Later