| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-07 | |||
| 18:08:34 | jaypipes | dansmith: thx | |
| 18:33:39 | openstackgerrit | Merged openstack/nova master: FUP for test_reshape https://review.openstack.org/641482 | |
| 18:33:57 | openstackgerrit | Merged openstack/nova master: Update --max-rows parameter description for archive_deleted_rows https://review.openstack.org/606995 | |
| 18:34:07 | openstackgerrit | Merged openstack/nova master: tests: Create PCI tests for NUMA'y tests https://review.openstack.org/640826 | |
| 18:41:54 | mnaser | i'm dealing with a sorts of 'usability' issue. booting a cinder volume from a glance image (where the image is stored on local disk) | |
| 18:42:08 | mnaser | cinder eventually creates the image, but it takes around 15 minutes because it has to download+upload | |
| 18:42:35 | mnaser | nova times out waiting for the volume, even though it's in `downloading` state | |
| 18:42:39 | smcginnis | mnaser: Do you have image cache enabled? | |
| 18:42:50 | smcginnis | That will help after the first one at least. | |
| 18:43:01 | mnaser | smcginnis: interesting, i don't think image cache is enabled for this deployment, but yes, that's a good idea | |
| 18:43:19 | smcginnis | Creating the boot volume via Nova timing out is a known issue of sorts. | |
| 18:43:27 | smcginnis | You need to tweak the timeouts if you do it that way. | |
| 18:43:36 | mnaser | the type of user is one who refuses to create a volume and wait for it to then attach it is the one i'm dealing with :) | |
| 18:43:39 | smcginnis | Really recommended to create the volume first, then boot from it. | |
| 18:43:51 | mnaser | "it should just work <insert a ton of exclamation marks>" | |
| 18:44:16 | mriedem | mnaser: sure, just tell them to pre-cache their images :) | |
| 18:44:19 | smcginnis | I've actually been meaning to see how difficult it would be to just update horizon to look like it's the one path, but do the two separate steps under the covers to make it less likely to time out. | |
| 18:44:55 | mnaser | i'm just trying to see if we can have a better way in terms of usability, if cinder is reporting that its' still 'downloading' really | |
| 18:45:15 | mnaser | also, another fun thing is that it seems like it doesn't properly clean up as it tries to request a delete on a volume that's in downloading process | |
| 18:45:26 | mriedem | mnaser: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.block_device_allocate_retries and https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.block_device_allocate_retries_interval are the related nova-compute options | |
| 18:45:27 | mnaser | so seems like volumes end in error downloading state or something along those lines, but i haven't dug that far yet | |
| 18:45:56 | mriedem | it's this code https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1308 | |
| 18:46:06 | smcginnis | Hmm, error downloading sounds like a bigger issue to address first. | |
| 18:46:43 | mnaser | i just want to avoid changing the timeout to '15 minutes' because they will bring bigger images and then ask me to bump the timeout to '30 minutes' lols | |
| 18:46:51 | mnaser | smcginnis: yeah, let me see if i have tracebacks for that too | |
| 18:47:06 | mriedem | mnaser: this is the code that does the waiting and cleanup https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L689 | |
| 18:47:17 | mriedem | nova won't delete the volume if delete_on_termination is False | |
| 18:47:20 | mriedem | which it is by default i think | |
| 18:51:21 | mnaser | i guess the cleanest way is image cache *but* that does involve booting it once still | |
| 18:51:38 | mnaser | we didn't have this issue when using cinder backend for glance | |
| 18:51:57 | mnaser | but then we have the issue with glance buffering a whole image to disk and then pushing to array | |
| 18:52:25 | smcginnis | I think I've been told if you use ceph it helps a lot. | |
| 18:52:36 | mnaser | yes we're hoping to kill off their dell ps array | |
| 18:52:44 | mnaser | ceph is wonderful for all of this | |
| 18:52:55 | smcginnis | That might be good anyway. PS is no longer going to be supported. | |
| 18:53:11 | mnaser | good, then we can force them to get off | |
| 18:53:12 | mnaser | :-P | |
| 18:53:23 | smcginnis | ;) | |
| 18:58:40 | openstackgerrit | Merged openstack/nova master: Move legacy-grenade-dsvm-neutron-multinode-live-migration in-tree https://review.openstack.org/640181 | |
| 19:01:05 | openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | |
| 19:07:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Clean up block_device_allocate_retries config option help https://review.openstack.org/641759 | |
| 19:07:27 | mriedem | mnaser: i'm having a problem letting shitty docs slide lately ^ | |
| 19:08:36 | mnaser | mriedem: i like that | |
| 19:08:41 | mnaser | that's a lot more clear | |
| 19:10:57 | openstackgerrit | Merged openstack/nova master: Make nova-grenade-live-migration voting and gating https://review.openstack.org/640182 | |
| 19:17:58 | mnaser | ok i've got this really strange breakage inside OSA that seems to happen within nova | |
| 19:18:04 | mnaser | http://logs.openstack.org/08/641308/2/check/openstack-ansible-deploy-aio_lxc-centos-7/2485c9a/logs/openstack/aio1-nova/nova-compute.log.txt.gz#_2019-03-06_12_35_49_535 | |
| 19:18:10 | mnaser | "Neutron Reported failure on event network-vif-plugged-4095592d-730d-4a32-82b7-1ec3133f5862 for instance a43072b3-43e5-4fce-ae32-c11510b569fc: NovaException: In shutdown, no new events can be scheduled" | |
| 19:18:29 | mnaser | seems to me that the ` NovaException: In shutdown, no new events can be scheduled` portion is actually coming from `prepare_for_instance_event` | |
| 19:18:42 | mnaser | so the error is a bit misleading too | |
| 19:18:51 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n317 | |
| 19:19:07 | mnaser | yeah, i dug that out, i need a faster way to link from vscode :< | |
| 19:19:23 | mnaser | i'm not sure what or why self._events is, and why it's set to `None` | |
| 19:19:31 | mnaser | the port seems to be plugging fine from the neutron side of things, according to the logs | |
| 19:19:45 | mriedem | self._events = None generally means nova-compute is stopping | |
| 19:20:05 | mnaser | strange, it doesn't seeeeeeem to be in those logs? you can see it continue to do things | |
| 19:20:09 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n397 | |
| 19:20:33 | mriedem | the only thing that calls that is cleanup_host http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n1236 | |
| 19:20:45 | dansmith | that message is exactly what is supposed to happen if compute is shutting down | |
| 19:20:55 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/service.py#n286 | |
| 19:21:16 | mnaser | that's so strange, interestingly enough, the logs don't show anything about cleaning up | |
| 19:21:22 | mnaser | or any of the "popping stuff out of event queue" messages | |
| 19:21:50 | mnaser | we did recently merge a change where we had vif_plugging_is_fatal default to true, so this might have exposed something (and it was always failing?! i hope not..) | |
| 19:22:12 | dansmith | vif plugging fatal should be set to true | |
| 19:22:18 | mnaser | sorry, i mean we set it to false | |
| 19:22:22 | dansmith | ack okay | |
| 19:22:34 | dansmith | wait, you mean you had been setting it to false, and recently fixed that yeah? | |
| 19:22:39 | mnaser | yeah | |
| 19:22:44 | dansmith | either way, that won't affect the incoming events, | |
| 19:22:49 | dansmith | just whether or not we wait for them | |
| 19:22:58 | dansmith | and what you show above is just event delivery | |
| 19:23:00 | mnaser | well, if something was broken before, we probably just exposed it (in osa or whatever) | |
| 19:23:13 | mnaser | we might have been just blindly plugging ports and it just worked™ -- this just exposed it | |
| 19:23:29 | mnaser | the interesting thing is i dont see "Canceling in-flight event" in the logs anywhere | |
| 19:23:42 | mnaser | but i guess the blocking new events might do it there | |
| 19:23:42 | mriedem | https://review.openstack.org/#/c/420026/ | |
| 19:23:47 | mriedem | ^ is it this? | |
| 19:23:58 | mriedem | did something sighup the service? | |
| 19:23:59 | dansmith | mnaser: you wouldn't if nothing is waiting when the shutdown starts | |
| 19:24:08 | dansmith | mriedem: see my comments/ | |
| 19:24:15 | dansmith | mriedem: we don't call cleanup_host on sighup | |
| 19:24:28 | mnaser | let me check systemd logs | |
| 19:24:59 | mriedem | well i'm all out of ideas :P | |
| 19:25:02 | dansmith | if we did we'd be tearing down rpc and dropping things all over the place, which would be a pretty big problem for what sighup is supposed to do | |
| 19:25:38 | mriedem | mnaser: are you hitting this consistently in osa ci? because a simple debug patch should tell us what is triggering the stop | |
| 19:25:40 | mnaser | http://logs.openstack.org/08/641308/2/check/openstack-ansible-deploy-aio_lxc-centos-7/2485c9a/logs/host/messages.txt.gz#_Mar__6_12_12_43 we reload here, but that's almost 23 minutes before it happened | |
| 19:25:51 | mnaser | yes, we're perma-borked right now | |
| 19:26:14 | dansmith | mnaser: are you maybe starting to shut down nova services before tests have finished or something? | |
| 19:26:17 | mnaser | that's the latest reference to a the nova-compute service | |
| 19:26:30 | dansmith | or maybe tests are done but instances are still coming up or something? | |
| 19:26:48 | mnaser | system logs would have shown us shutting down the service (it's a systemd unit) | |
| 19:27:05 | mnaser | it is possible that a reload might put the nova-compute in a weird state? | |
| 19:27:06 | dansmith | well, if you're using systemd to shut it down, sure | |
| 19:27:24 | dansmith | mnaser: shouldn't and the code does very little on sighup | |
| 19:29:01 | mnaser | oh thats weird | |
| 19:29:06 | mnaser | the change mriedem linked is missing in master | |
| 19:29:12 | mnaser | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L548 | |
| 19:29:12 | dansmith | it's unmerged | |
| 19:29:15 | dansmith | because it's bad | |
| 19:29:17 | mnaser | ah oops | |