| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-07 | |||
| 17:45:26 | sean-k-mooney | oh ok i missed that then. cool | |
| 17:46:06 | dansmith | adding it in stein would tell them they need to do something they can't do until they deploy stein code | |
| 17:46:45 | sean-k-mooney | right because we are only adding the column in stien | |
| 17:49:10 | jaypipes | dansmith, mriedem: bottom 3 +W. still plowing through the nova-manage migration in https://review.openstack.org/#/c/633351... | |
| 17:51:17 | openstackgerrit | Matthew Booth proposed openstack/nova master: Allow utime call to fail on qcow2 image base file https://review.openstack.org/641315 | |
| 17:51:42 | mriedem | jaypipes: don't make me get you a mr plow jacket | |
| 17:51:46 | mriedem | so you can strut around | |
| 17:52:07 | mriedem | https://www.youtube.com/watch?v=fhW3OyifFBE | |
| 17:54:24 | dansmith | mriedem: how would you communicate with the other members of the tribe if you didn't have the simpsons and every snippet documented on youtube? | |
| 17:54:34 | mriedem | grunts | |
| 17:54:43 | mriedem | flute sounds | |
| 17:55:05 | jaypipes | heh | |
| 17:55:44 | mriedem | jaypipes: so i think i'll abstain from proposing a unified limits session, it's out of my ability to lead that | |
| 17:57:28 | jaypipes | mriedem: I can participate in such a session. | |
| 17:58:04 | mriedem | i'm going to go participate in a bowl of chili | |
| 17:58:45 | jaypipes | :) | |
| 17:59:04 | cdent | a forum session with a chili bakeoff would be pretty cool | |
| 18:00:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Fix nits in remote console guide https://review.openstack.org/641098 | |
| 18:00:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Rework all things metadata'y https://review.openstack.org/640730 | |
| 18:00:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Rename 'configuration drive' to 'config drive' https://review.openstack.org/641723 | |
| 18:03:59 | jaypipes | mriedem_afk, dansmith: please see my question on https://review.openstack.org/#/c/633351/. it's the only part of that migration I don't get... | |
| 18:06:06 | dansmith | jaypipes: replied | |
| 18:08:34 | jaypipes | dansmith: thx | |
| 18:33:39 | openstackgerrit | Merged openstack/nova master: FUP for test_reshape https://review.openstack.org/641482 | |
| 18:33:57 | openstackgerrit | Merged openstack/nova master: Update --max-rows parameter description for archive_deleted_rows https://review.openstack.org/606995 | |
| 18:34:07 | openstackgerrit | Merged openstack/nova master: tests: Create PCI tests for NUMA'y tests https://review.openstack.org/640826 | |
| 18:41:54 | mnaser | i'm dealing with a sorts of 'usability' issue. booting a cinder volume from a glance image (where the image is stored on local disk) | |
| 18:42:08 | mnaser | cinder eventually creates the image, but it takes around 15 minutes because it has to download+upload | |
| 18:42:35 | mnaser | nova times out waiting for the volume, even though it's in `downloading` state | |
| 18:42:39 | smcginnis | mnaser: Do you have image cache enabled? | |
| 18:42:50 | smcginnis | That will help after the first one at least. | |
| 18:43:01 | mnaser | smcginnis: interesting, i don't think image cache is enabled for this deployment, but yes, that's a good idea | |
| 18:43:19 | smcginnis | Creating the boot volume via Nova timing out is a known issue of sorts. | |
| 18:43:27 | smcginnis | You need to tweak the timeouts if you do it that way. | |
| 18:43:36 | mnaser | the type of user is one who refuses to create a volume and wait for it to then attach it is the one i'm dealing with :) | |
| 18:43:39 | smcginnis | Really recommended to create the volume first, then boot from it. | |
| 18:43:51 | mnaser | "it should just work <insert a ton of exclamation marks>" | |
| 18:44:16 | mriedem | mnaser: sure, just tell them to pre-cache their images :) | |
| 18:44:19 | smcginnis | I've actually been meaning to see how difficult it would be to just update horizon to look like it's the one path, but do the two separate steps under the covers to make it less likely to time out. | |
| 18:44:55 | mnaser | i'm just trying to see if we can have a better way in terms of usability, if cinder is reporting that its' still 'downloading' really | |
| 18:45:15 | mnaser | also, another fun thing is that it seems like it doesn't properly clean up as it tries to request a delete on a volume that's in downloading process | |
| 18:45:26 | mriedem | mnaser: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.block_device_allocate_retries and https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.block_device_allocate_retries_interval are the related nova-compute options | |
| 18:45:27 | mnaser | so seems like volumes end in error downloading state or something along those lines, but i haven't dug that far yet | |
| 18:45:56 | mriedem | it's this code https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1308 | |
| 18:46:06 | smcginnis | Hmm, error downloading sounds like a bigger issue to address first. | |
| 18:46:43 | mnaser | i just want to avoid changing the timeout to '15 minutes' because they will bring bigger images and then ask me to bump the timeout to '30 minutes' lols | |
| 18:46:51 | mnaser | smcginnis: yeah, let me see if i have tracebacks for that too | |
| 18:47:06 | mriedem | mnaser: this is the code that does the waiting and cleanup https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L689 | |
| 18:47:17 | mriedem | nova won't delete the volume if delete_on_termination is False | |
| 18:47:20 | mriedem | which it is by default i think | |
| 18:51:21 | mnaser | i guess the cleanest way is image cache *but* that does involve booting it once still | |
| 18:51:38 | mnaser | we didn't have this issue when using cinder backend for glance | |
| 18:51:57 | mnaser | but then we have the issue with glance buffering a whole image to disk and then pushing to array | |
| 18:52:25 | smcginnis | I think I've been told if you use ceph it helps a lot. | |
| 18:52:36 | mnaser | yes we're hoping to kill off their dell ps array | |
| 18:52:44 | mnaser | ceph is wonderful for all of this | |
| 18:52:55 | smcginnis | That might be good anyway. PS is no longer going to be supported. | |
| 18:53:11 | mnaser | good, then we can force them to get off | |
| 18:53:12 | mnaser | :-P | |
| 18:53:23 | smcginnis | ;) | |
| 18:58:40 | openstackgerrit | Merged openstack/nova master: Move legacy-grenade-dsvm-neutron-multinode-live-migration in-tree https://review.openstack.org/640181 | |
| 19:01:05 | openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | |
| 19:07:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Clean up block_device_allocate_retries config option help https://review.openstack.org/641759 | |
| 19:07:27 | mriedem | mnaser: i'm having a problem letting shitty docs slide lately ^ | |
| 19:08:36 | mnaser | mriedem: i like that | |
| 19:08:41 | mnaser | that's a lot more clear | |
| 19:10:57 | openstackgerrit | Merged openstack/nova master: Make nova-grenade-live-migration voting and gating https://review.openstack.org/640182 | |
| 19:17:58 | mnaser | ok i've got this really strange breakage inside OSA that seems to happen within nova | |
| 19:18:04 | mnaser | http://logs.openstack.org/08/641308/2/check/openstack-ansible-deploy-aio_lxc-centos-7/2485c9a/logs/openstack/aio1-nova/nova-compute.log.txt.gz#_2019-03-06_12_35_49_535 | |
| 19:18:10 | mnaser | "Neutron Reported failure on event network-vif-plugged-4095592d-730d-4a32-82b7-1ec3133f5862 for instance a43072b3-43e5-4fce-ae32-c11510b569fc: NovaException: In shutdown, no new events can be scheduled" | |
| 19:18:29 | mnaser | seems to me that the ` NovaException: In shutdown, no new events can be scheduled` portion is actually coming from `prepare_for_instance_event` | |
| 19:18:42 | mnaser | so the error is a bit misleading too | |
| 19:18:51 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n317 | |
| 19:19:07 | mnaser | yeah, i dug that out, i need a faster way to link from vscode :< | |
| 19:19:23 | mnaser | i'm not sure what or why self._events is, and why it's set to `None` | |
| 19:19:31 | mnaser | the port seems to be plugging fine from the neutron side of things, according to the logs | |
| 19:19:45 | mriedem | self._events = None generally means nova-compute is stopping | |
| 19:20:05 | mnaser | strange, it doesn't seeeeeeem to be in those logs? you can see it continue to do things | |
| 19:20:09 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n397 | |
| 19:20:33 | mriedem | the only thing that calls that is cleanup_host http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/manager.py#n1236 | |
| 19:20:45 | dansmith | that message is exactly what is supposed to happen if compute is shutting down | |
| 19:20:55 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/service.py#n286 | |
| 19:21:16 | mnaser | that's so strange, interestingly enough, the logs don't show anything about cleaning up | |
| 19:21:22 | mnaser | or any of the "popping stuff out of event queue" messages | |
| 19:21:50 | mnaser | we did recently merge a change where we had vif_plugging_is_fatal default to true, so this might have exposed something (and it was always failing?! i hope not..) | |
| 19:22:12 | dansmith | vif plugging fatal should be set to true | |
| 19:22:18 | mnaser | sorry, i mean we set it to false | |
| 19:22:22 | dansmith | ack okay | |
| 19:22:34 | dansmith | wait, you mean you had been setting it to false, and recently fixed that yeah? | |
| 19:22:39 | mnaser | yeah | |
| 19:22:44 | dansmith | either way, that won't affect the incoming events, | |
| 19:22:49 | dansmith | just whether or not we wait for them | |
| 19:22:58 | dansmith | and what you show above is just event delivery | |
| 19:23:00 | mnaser | well, if something was broken before, we probably just exposed it (in osa or whatever) | |
| 19:23:13 | mnaser | we might have been just blindly plugging ports and it just worked™ -- this just exposed it | |
| 19:23:29 | mnaser | the interesting thing is i dont see "Canceling in-flight event" in the logs anywhere | |
| 19:23:42 | mnaser | but i guess the blocking new events might do it there | |
| 19:23:42 | mriedem | https://review.openstack.org/#/c/420026/ | |
| 19:23:47 | mriedem | ^ is it this? | |
| 19:23:58 | mriedem | did something sighup the service? | |