| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-13 | |||
| 16:06:33 | mriedem | but i thought n-cpu went last | |
| 16:06:40 | dansmith | the error says no connection parameter, right? | |
| 16:06:59 | dansmith | oh, this is a RemoteError btw | |
| 16:07:24 | dansmith | so the cell conductor doesn't have it, | |
| 16:07:31 | dansmith | but cpu does, so it tried a remotable all_cells call maybe? | |
| 16:07:39 | dansmith | no, that doesn't make sense | |
| 16:08:00 | mriedem | right nova_cell1.conf (conductor) doesn't have api_database http://logs.openstack.org/82/624982/1/check/nova-next/4d4ec67/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 16:08:27 | dansmith | ph. | |
| 16:08:29 | dansmith | yeah | |
| 16:08:37 | dansmith | it's doing load_cells, which is where the remotable comes in | |
| 16:08:42 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L505 | |
| 16:08:50 | dansmith | so yeah just nuking the cpu config api database should fix it I think | |
| 16:08:57 | mriedem | compute manager init creates compute.API which creates the compute rpcapi client which checks the api db conf | |
| 16:09:07 | dansmith | File "/opt/stack/new/nova/nova/context.py", line 476, in load_cells | |
| 16:09:16 | mriedem | https://github.com/openstack-dev/devstack/commit/7d0003ef7ed2c2609ffe6782e0ce6147f9c9a3bf#diff-665a24457e945ac31372cf63d00a4080 | |
| 16:09:18 | mriedem | just need that | |
| 16:09:19 | dansmith | right, which ends up calling into load_cells, which does the remotable I think | |
| 16:09:22 | dansmith | yar | |
| 16:09:25 | mriedem | i think before rocky, devstack didn't use upgrade_levels=auto | |
| 16:09:34 | dansmith | ack | |
| 16:10:08 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Make compute rpcapi version calculation check all cells https://review.openstack.org/624982 | |
| 16:25:22 | mriedem | dansmith: hmm, thinking about that... | |
| 16:25:34 | mriedem | people are setting api db in cell configs, | |
| 16:25:37 | mriedem | so they can do the up-call stuff | |
| 16:25:54 | mriedem | so when nova-compute starts up in cell1, it could be determining it's min version from computes in cell50 | |
| 16:25:55 | dansmith | as long as it's the same on the compute and corresponding conductor it'll befine | |
| 16:25:56 | mriedem | in cerns case | |
| 16:26:06 | dansmith | oh | |
| 16:26:21 | dansmith | I see what you mean, but that probably isn't terrible | |
| 16:26:40 | dansmith | if you're giving the cell visibility to the whole deployment, then .. you get the whole deployment's min version | |
| 16:26:52 | dansmith | this is exactly why I *don't* want to make this a common practice, fwiw, | |
| 16:27:05 | dansmith | because now you're talking about teaching the cells about their place in the world and that gets icky | |
| 16:28:04 | mriedem | i just don't know if it could throw anyone off, i.e. upgrade reno or anything, | |
| 16:28:11 | mriedem | but if you're doing upgrade_levels=auto, | |
| 16:28:19 | mriedem | you're opting into the system figuring this out | |
| 16:28:25 | mriedem | and can pin manually if you don't trust the system | |
| 16:28:42 | mriedem | let's pretend i never mentioned this | |
| 16:28:59 | dansmith | yep | |
| 16:35:22 | cdent | mriedem: I've added some links to https://bugs.launchpad.net/openstack-gate/+bug/1808063 for the recent failure that you rechecked. g-api is having a stackoverflow | |
| 16:35:22 | openstack | Launchpad bug 1808063 in OpenStack-Gate "glanceclient.exc.HTTPBadGateway: 502 Proxy Error during server snapshot" [Undecided,Confirmed] | |
| 16:38:45 | mriedem | oh nice | |
| 16:38:46 | mriedem | http://logs.openstack.org/41/617941/31/gate/grenade-py3/7909f30/logs/screen-g-api.txt.gz#_Dec_13_14_43_21_414511 | |
| 16:38:50 | mriedem | Dec 13 14:43:21.414511 ubuntu-xenial-rax-ord-0001176665 devstack@g-api.service[6990]: File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 926 in _heartbeat_thread_job | |
| 16:39:03 | mriedem | dansmith: look familiar? ^ | |
| 16:39:10 | cdent | and a second comment | |
| 16:39:25 | mriedem | File "/opt/stack/new/glance/glance/quota/__init__.py", line 168 in __getattr__ looks like infinite loop? | |
| 16:39:33 | cdent | eyes | |
| 16:39:35 | cdent | yes | |
| 16:40:03 | mriedem | https://github.com/openstack/glance/commit/cb45edf5c81f7d09c9ef0b88d40d56b4750beb10#diff-5afd3797ee6a482f2c0d4862140f7014 was the last change in there but awhile ago | |
| 16:40:42 | mriedem | https://github.com/openstack/glance/blob/master/glance/quota/__init__.py#L168 | |
| 16:44:54 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Only warn about not having computes nodes once in rpcapi https://review.openstack.org/625051 | |
| 17:02:12 | melwitt | mriedem: +1 to flushing queens and pike backports and releasing before the holiday break. I assumed not rocky bc we did that semi recently but were you thinking rocky too? I'm sure a lot has landed since last time | |
| 17:02:37 | melwitt | I will help flush | |
| 17:11:22 | mriedem | melwitt: yes otherwise we could release things in pike/queens that are not yet released in rocky | |
| 17:11:31 | mriedem | generally need to snowball these from newest to oldest branch | |
| 17:11:44 | melwitt | good point, I should have thought of that. thanks | |
| 17:12:07 | openstackgerrit | Merged openstack/nova stable/rocky: Handle tags in _bury_in_cell0 https://review.openstack.org/624813 | |
| 17:14:31 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Make compute rpcapi version calculation check all cells https://review.openstack.org/625060 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: Propagate exception message from _prep_block_device https://review.openstack.org/624832 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: Add method to generate device names universally https://review.openstack.org/573066 | |
| 17:35:59 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Add configuration of maximum disk devices to attach https://review.openstack.org/616777 | |
| 17:40:47 | colby_ | Hey Eveyrone. We just upgraded to queeens. Im seeing some weird issues when creating an instance from an already created volume. Im seeing this error: | |
| 17:40:47 | colby_ | Build of instance aborted: Nova does not support Cinder API version 2. The catalog is set to use v3 cinder as default, and in the nova config we have this set for cinder: catalog_info=volumev3:cinderv3:internalURL | |
| 17:41:06 | colby_ | we have a volumev3 in the catalog as well that points to the v3 api | |
| 17:55:20 | melwitt | colby_: is it possible you have [cinder]endpoint_template set, which will override catalog_info? | |
| 18:01:55 | cezary_zukowski | Hey, how to simulate/mimic a network disruption occuring between a Nova instance/server and Cinder (volume/storage)? Does detaching a volume from instance manually would be a sensible approach/substitute? | |
| 18:01:57 | mriedem | that would be my guess, endpoint_template is set | |
| 18:02:20 | cezary_zukowski | Command for detaching: nova volume-detach <instance_id> <volume_id> | |
| 18:02:48 | cezary_zukowski | mriedem: What do you mean? | |
| 18:03:38 | colby_ | melwitt: no its not set in the config anywhere | |
| 18:03:46 | mriedem | cezary_zukowski: was answering a different question | |
| 18:03:56 | cezary_zukowski | mriedem: ah, right | |
| 18:04:29 | mriedem | colby_: do you have this? https://github.com/openstack/nova/commit/c7d87f6691bbc39daca918729b769cc2d2000abd | |
| 18:04:45 | mriedem | that's in 17.0.6 | |
| 18:04:50 | mriedem | if you're queens ga you wouldn't have that | |
| 18:05:13 | mriedem | not sure if that's your problem though | |
| 18:17:42 | melwitt | colby_: aside from all of that, I would just take a look at your keystone catalog again and double check there's no /v2.0 for nova (should be /v2.1) and no /v2 for cinder | |
| 18:18:33 | melwitt | because the only other thing I see the code doing to determine the cinder version is parsing the version from the endpoint found in the catalog | |
| 18:18:45 | colby_ | oh it has the wrong config on the actual compute nodes. Thanks that helped me narrow it down. Ill switch that config there. Apparently it just got switched on the controller services | |
| 18:20:14 | melwitt | ok, cool | |
| 18:45:35 | mpiwowarczy | hello | |
| 18:46:36 | mpiwowarczy | can I have a question to nova deprecated personality parameter? | |
| 18:47:14 | mpiwowarczy | seems it will be removed in microversion 2.57 | |
| 18:47:42 | mpiwowarczy | but can not found when 2.57 will be used? | |
| 18:47:55 | mpiwowarczy | for Rocky release it was microversion 2.27 | |
| 18:48:06 | mpiwowarczy | which version will be used in Stein? | |
| 18:49:01 | mpiwowarczy | or is there any fixed date when 2.57 version will be dropped? | |
| 18:54:22 | artom | mpiwowarczy, microversions are never dropped, we still support 2.1 if the client asks for it | |
| 18:54:34 | artom | mpiwowarczy, in terms of releases, have a look at https://docs.openstack.org/nova/latest/reference/api-microversion-history.html | |
| 18:56:49 | mpiwowarczy | artom: thanks for the answer | |
| 18:56:56 | mpiwowarczy | so when personality files (files injection) feature will be dropped? | |
| 18:58:11 | mpiwowarczy | will be this supported in Stein release? | |
| 19:01:35 | artom | mpiwowarczy, it'll never be dropped per se | |
| 19:02:05 | artom | mpiwowarczy, you can keep using it in Stein if the client asks for a microversion < 2.57 | |
| 19:03:49 | mpiwowarczy | ok, thanks for your help | |
| 19:07:37 | melwitt | mpiwowarczy: it's dropped in a sense that if/when you find yourself needing to request microversion >= 2.57 when you call the REST API to get a different feature/addition, you will not be able to use personality files at the same time/in the same call | |
| 19:08:40 | mpiwowarczy | melwitt: I see | |
| 19:10:15 | mpiwowarczy | do you intend to maintain all versions, starting from 2.1, in all future OpenStack releases? | |
| 19:11:22 | mpiwowarczy | day by day it is more difficult to maintain :) | |
| 19:18:33 | mriedem | what is more difficult to maintain? writing apps against the API? or maintaining the various microversions in the API? | |