| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-13 | |||
| 15:10:10 | gibi | sean-k-mooney: the leads to a situation described in the patch https://review.openstack.org/#/c/623543/ | |
| 15:11:20 | gibi | sean-k-mooney: and we discussed a bit about this in Berlin with adrianc | |
| 15:13:22 | sean-k-mooney | gibi: ill be leaveing soon but ill take a look tomorow. i can try and test it next week. | |
| 15:13:54 | gibi | sean-k-mooney: no rush, that patch is still WIP | |
| 15:14:28 | gibi | sean-k-mooney: it is more like that the problem statement might be relevant for your live migration work | |
| 15:19:09 | openstackgerrit | Jose Castro Leon proposed openstack/nova master: Fix destination_type attribute in the bdm_v2 documentation https://review.openstack.org/625020 | |
| 15:36:08 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810 | |
| 15:36:09 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657 | |
| 15:41:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740 | |
| 15:41:50 | mriedem | d'oh | |
| 15:42:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740 | |
| 15:42:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add python 3.7 unit and functional tox jobs https://review.openstack.org/624055 | |
| 15:42:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add py36/py37 functional jobs to the experimental queue https://review.openstack.org/624826 | |
| 15:42:45 | mriedem | gibi: i was rebasing and fixing that at the same time | |
| 15:43:00 | gibi | mriedem: no problemo | |
| 15:44:17 | gibi | stephenfin: could you look at it? ^^ | |
| 15:45:34 | gibi | stephenfin: mriedem and I agreed on the solution | |
| 15:59:55 | mriedem | ooo goody http://logs.openstack.org/82/624982/1/check/nova-next/4d4ec67/logs/screen-n-cpu.txt.gz | |
| 16:00:51 | dansmith | looks fine | |
| 16:03:38 | mriedem | dansmith: that's the backport of your multi-cell n-api rpc version check | |
| 16:03:44 | mriedem | the n-cpu failures there have to be unrelated | |
| 16:04:23 | mriedem | i thought we collected mysql logs but maybe not anymore | |
| 16:04:29 | dansmith | hmm, it shouldn't check all cells if the api connection isn't set | |
| 16:04:38 | mriedem | this is n-cpu failing to start | |
| 16:05:06 | mriedem | File "/opt/stack/new/nova/nova/objects/cell_mapping.py", line 257, in get_all\n | |
| 16:05:06 | mriedem | wtf | |
| 16:05:27 | mriedem | oooo | |
| 16:05:42 | mriedem | http://logs.openstack.org/82/624982/1/check/nova-next/4d4ec67/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 16:05:51 | mriedem | we still have the api db conf set in rocky devstack | |
| 16:06:04 | dansmith | but.. then it shouldn't fail right? | |
| 16:06:23 | mriedem | unless it's starting n-cpu before the api db is up? | |
| 16:06:33 | mriedem | but i thought n-cpu went last | |
| 16:06:40 | dansmith | the error says no connection parameter, right? | |
| 16:06:59 | dansmith | oh, this is a RemoteError btw | |
| 16:07:24 | dansmith | so the cell conductor doesn't have it, | |
| 16:07:31 | dansmith | but cpu does, so it tried a remotable all_cells call maybe? | |
| 16:07:39 | dansmith | no, that doesn't make sense | |
| 16:08:00 | mriedem | right nova_cell1.conf (conductor) doesn't have api_database http://logs.openstack.org/82/624982/1/check/nova-next/4d4ec67/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 16:08:27 | dansmith | ph. | |
| 16:08:29 | dansmith | yeah | |
| 16:08:37 | dansmith | it's doing load_cells, which is where the remotable comes in | |
| 16:08:42 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L505 | |
| 16:08:50 | dansmith | so yeah just nuking the cpu config api database should fix it I think | |
| 16:08:57 | mriedem | compute manager init creates compute.API which creates the compute rpcapi client which checks the api db conf | |
| 16:09:07 | dansmith | File "/opt/stack/new/nova/nova/context.py", line 476, in load_cells | |
| 16:09:16 | mriedem | https://github.com/openstack-dev/devstack/commit/7d0003ef7ed2c2609ffe6782e0ce6147f9c9a3bf#diff-665a24457e945ac31372cf63d00a4080 | |
| 16:09:18 | mriedem | just need that | |
| 16:09:19 | dansmith | right, which ends up calling into load_cells, which does the remotable I think | |
| 16:09:22 | dansmith | yar | |
| 16:09:25 | mriedem | i think before rocky, devstack didn't use upgrade_levels=auto | |
| 16:09:34 | dansmith | ack | |
| 16:10:08 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Make compute rpcapi version calculation check all cells https://review.openstack.org/624982 | |
| 16:25:22 | mriedem | dansmith: hmm, thinking about that... | |
| 16:25:34 | mriedem | people are setting api db in cell configs, | |
| 16:25:37 | mriedem | so they can do the up-call stuff | |
| 16:25:54 | mriedem | so when nova-compute starts up in cell1, it could be determining it's min version from computes in cell50 | |
| 16:25:55 | dansmith | as long as it's the same on the compute and corresponding conductor it'll befine | |
| 16:25:56 | mriedem | in cerns case | |
| 16:26:06 | dansmith | oh | |
| 16:26:21 | dansmith | I see what you mean, but that probably isn't terrible | |
| 16:26:40 | dansmith | if you're giving the cell visibility to the whole deployment, then .. you get the whole deployment's min version | |
| 16:26:52 | dansmith | this is exactly why I *don't* want to make this a common practice, fwiw, | |
| 16:27:05 | dansmith | because now you're talking about teaching the cells about their place in the world and that gets icky | |
| 16:28:04 | mriedem | i just don't know if it could throw anyone off, i.e. upgrade reno or anything, | |
| 16:28:11 | mriedem | but if you're doing upgrade_levels=auto, | |
| 16:28:19 | mriedem | you're opting into the system figuring this out | |
| 16:28:25 | mriedem | and can pin manually if you don't trust the system | |
| 16:28:42 | mriedem | let's pretend i never mentioned this | |
| 16:28:59 | dansmith | yep | |
| 16:35:22 | cdent | mriedem: I've added some links to https://bugs.launchpad.net/openstack-gate/+bug/1808063 for the recent failure that you rechecked. g-api is having a stackoverflow | |
| 16:35:22 | openstack | Launchpad bug 1808063 in OpenStack-Gate "glanceclient.exc.HTTPBadGateway: 502 Proxy Error during server snapshot" [Undecided,Confirmed] | |
| 16:38:45 | mriedem | oh nice | |
| 16:38:46 | mriedem | http://logs.openstack.org/41/617941/31/gate/grenade-py3/7909f30/logs/screen-g-api.txt.gz#_Dec_13_14_43_21_414511 | |
| 16:38:50 | mriedem | Dec 13 14:43:21.414511 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]: File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 926 in _heartbeat_thread_job | |
| 16:39:03 | mriedem | dansmith: look familiar? ^ | |
| 16:39:10 | cdent | and a second comment | |
| 16:39:25 | mriedem | File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__ looks like infinite loop? | |
| 16:39:33 | cdent | eyes | |
| 16:39:35 | cdent | yes | |
| 16:40:03 | mriedem | https://github.com/openstack/glance/commit/cb45edf5c81f7d09c9ef0b88d40d56b4750beb10#diff-5afd3797ee6a482f2c0d4862140f7014 was the last change in there but awhile ago | |
| 16:40:42 | mriedem | https://github.com/openstack/glance/blob/master/glance/quota/__init__.py#L168 | |
| 16:44:54 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Only warn about not having computes nodes once in rpcapi https://review.openstack.org/625051 | |
| 17:02:12 | melwitt | mriedem: +1 to flushing queens and pike backports and releasing before the holiday break. I assumed not rocky bc we did that semi recently but were you thinking rocky too? I'm sure a lot has landed since last time | |
| 17:02:37 | melwitt | I will help flush | |
| 17:11:22 | mriedem | melwitt: yes otherwise we could release things in pike/queens that are not yet released in rocky | |
| 17:11:31 | mriedem | generally need to snowball these from newest to oldest branch | |
| 17:11:44 | melwitt | good point, I should have thought of that. thanks | |
| 17:12:07 | openstackgerrit | Merged openstack/nova stable/rocky: Handle tags in _bury_in_cell0 https://review.openstack.org/624813 | |
| 17:14:31 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Make compute rpcapi version calculation check all cells https://review.openstack.org/625060 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: Propagate exception message from _prep_block_device https://review.openstack.org/624832 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: Add method to generate device names universally https://review.openstack.org/573066 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Add configuration of maximum disk devices to attach https://review.openstack.org/616777 | |
| 17:40:47 | colby_ | Hey Eveyrone. We just upgraded to queeens. Im seeing some weird issues when creating an instance from an already created volume. Im seeing this error: | |
| 17:40:47 | colby_ | Build of instance aborted: Nova does not support Cinder API version 2. The catalog is set to use v3 cinder as default, and in the nova config we have this set for cinder: catalog_info=volumev3:cinderv3:internalURL | |
| 17:41:06 | colby_ | we have a volumev3 in the catalog as well that points to the v3 api | |
| 17:55:20 | melwitt | colby_: is it possible you have [cinder]endpoint_template set, which will override catalog_info? | |
| 18:01:55 | cezary_zukowski | Hey, how to simulate/mimic a network disruption occuring between a Nova instance/server and Cinder (volume/storage)? Does detaching a volume from instance manually would be a sensible approach/substitute? | |
| 18:01:57 | mriedem | that would be my guess, endpoint_template is set | |
| 18:02:20 | cezary_zukowski | Command for detaching: nova volume-detach <instance_id> <volume_id> | |
| 18:02:48 | cezary_zukowski | mriedem: What do you mean? | |