Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-21
20:31:30 sean-k-mooney edmondsw: well it will if you are not useing postgres
20:31:45 edmondsw if you're not using postgres then you don't need psycopg2 :)
20:31:57 sean-k-mooney well that was basically my point
20:32:11 edmondsw but anyway... this is just a public service announcement
20:32:35 sean-k-mooney im not sure we can raise our lower constrants at this point can we?
20:32:51 efried for stein? It would take an emergency.
20:33:18 sean-k-mooney edmondsw: we have see this issue downstream by the way too but we just used 2.7 instead
20:33:29 mnaser https://bugs.launchpad.net/nova/+bug/1821244 im gonna work on this now. it should be a simple fix.
20:33:31 openstack Launchpad bug 1821244 in OpenStack Compute (nova) "Failed volume creation can result in invalid `connection_info` field" [Undecided,New]
20:33:49 edmondsw sean-k-mooney negating the purpose of lower-constraints
20:34:12 sean-k-mooney edmondsw: well this show up in older release too
20:34:59 sean-k-mooney the root casue was the presence of postgres 10.X that broke the compatiblity
20:35:17 sean-k-mooney if you use an older postgres it also works
20:35:47 sean-k-mooney but ya we should bump it but tommorow
20:38:46 sean-k-mooney efried: the lower constratins job should use nova lower constratin as the upper constratin
20:39:15 efried sean-k-mooney: even if it installs some other dep that has a higher lower constraint? That doesn't make sense.
20:39:21 sean-k-mooney pip transitive depencies via requirement.txt will not look at there upper or lower constratints
20:40:34 sean-k-mooney efried: it will fail to install if our constratin file say 2.0.0 and they request >= 3.0.0
20:41:59 sean-k-mooney upper/lower constratins.txt is an openstack convention so pip/setuptools does not know about them
20:42:36 sean-k-mooney pip just looks ate the requiremets.txt files
20:42:41 efried sure, but I would expect the l-c job to know about them.
20:42:56 efried ...about the lower constraints for the projects that have lower constraints.
20:43:10 sean-k-mooney the lower constratin job does pip install -r requriements.txt -c lower-constratints.txt
20:43:40 sean-k-mooney only one constratint file. nova is passed when installing the deps
20:43:51 efried mm, okay, then yeah, I guess it should only be observing my own l-c huh
20:46:14 sean-k-mooney for what its worth the job does look broken
20:46:21 sean-k-mooney http://logs.openstack.org/46/621646/20/check/openstack-tox-lower-constraints/8b49dc9/tox/lower-constraints-1.log
20:46:59 sean-k-mooney we are passing bot an upper-constratins.txt and lower-constraints.txt to pip
20:48:30 sean-k-mooney ok i see what going on
20:48:39 sean-k-mooney ill see if i can fix that
20:49:03 sean-k-mooney our install command include upper-constratints
20:51:24 efried sean-k-mooney: you mean L18 of tox.ini?
20:51:43 sean-k-mooney yes
20:52:09 sean-k-mooney im overriding hte install_command local to not do that in the lower-constraints env and ill see what hapens
20:52:20 sean-k-mooney i can see it endup using 2.7.7 in the gate
20:52:31 sean-k-mooney it should have been using 2.6.8 or whatever
20:53:13 sean-k-mooney well that failed instantly
20:53:16 efried so there's nothing tox-magical about the "lower-constraints" name like there is for pyXX that would be smart enough to use a fresh install_command?
20:53:41 efried failed instantly because a l-c is too low?
20:53:42 sean-k-mooney no lower-constraints is an openstack thing
20:53:49 efried okay.
20:54:06 sean-k-mooney it faild on the postgressql issue
20:54:10 efried so we've been borked for a while and real l-cs have been creeping past us.
20:54:29 efried edmondsw: Thanks for the find there!
20:54:33 sean-k-mooney ya for basicly ever i think looking at the git blame
20:54:50 sean-k-mooney the install command predate adding lower constratints i think
20:54:50 efried well, we've done quite a bit to tox.ini in the last release or two.
20:55:01 efried ...but not sure if those have been touched.
20:55:19 sean-k-mooney ill bump the lower constatin to 2.7 and push a patch shortly
20:55:30 efried sean-k-mooney: Well, that might not be the only transgressor
20:55:35 efried but I guess you'll find out :)
20:55:51 sean-k-mooney i was trying not to say that out loud but yes
20:57:20 sean-k-mooney i should proably file a bug for this. ya the next issue is the zvm sdk
20:57:57 sean-k-mooney i think i fixed this error actully it complingin about the python version support
20:58:45 sean-k-mooney i shoudl proably file a bug for this. shoudl this be a RC2 thing
21:00:57 openstackgerrit Merged openstack/nova-specs master: Re-propose count quota usage from placement https://review.openstack.org/645302
21:04:36 mnaser im struggling to trace the path of failure but i have a fix for this :\
21:07:02 openstackgerrit Mohammed Naser proposed openstack/nova master: wip: bdm: store empty object as connection_info by default https://review.openstack.org/645352
21:07:27 mnaser if someone has a few cycles to spare, i'm going to keep digging but i don't know the code that well to know why it's being saved with 'null'
21:08:23 sean-k-mooney mnaser: if the value is ever set to None it will be serialised to 'null'
21:08:55 sean-k-mooney i can try and take a look at it in a bit but i also dont know the bdm code that well
21:08:56 mnaser sean-k-mooney: yeah.. but i am trying to figure out why it would either be a) set to None or b) .get('connection_info') fails and returns None
21:09:30 mnaser oh interesting
21:09:34 sean-k-mooney if its trying to delete it before the block device mappings are constructed then that might be the issue
21:10:03 mnaser https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L284-L288
21:10:14 mnaser by default, the connection_info field defaults to 'NULL'
21:10:23 mnaser so that means self['connection_info'] = None ..
21:10:40 mriedem isn't the instance.host set before that though?
21:10:46 mriedem the instance_claim sets the instance.host
21:10:47 sean-k-mooney ya which just porpagates that default
21:10:51 mriedem so you shouldn't get a local delete flow in the API
21:11:09 mnaser mriedem: this is a case where instance.host = None so i think what happens is that the build aborts a few times on the compute host
21:11:17 mnaser and then when it fails to build it, instance.host goes to None
21:11:31 mnaser because cinder blows up when it tries to create a volume and you're out of quota
21:11:57 mnaser this isnt a case of instance.host = None because we're in cell0, it's because we failed to build the instance N times and gave up
21:12:12 mriedem anything failing in _prep_block_device automatically aborts the build
21:12:14 mriedem there is no reschedule
21:12:38 mriedem https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L2317
21:12:43 mriedem especially if you hit overquota
21:12:52 mnaser yeah im def hitting overquota because of the error logs
21:13:01 mnaser "Failed to create block device for instance due to exceeding volume related resource quota. Error: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 80G, quota is 6144G and 6080G has been consumed.: OverQuota: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 80G, quota is 6144G and 6080G has been consumed."
21:13:23 mriedem that should reset the instance.host to None here https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L2018
21:13:23 mnaser so that code is getting exercised
21:13:45 mriedem this is booting a volume from an image yeah?
21:13:46 mnaser which means a follow up delete will result in a local delete path?
21:13:47 mnaser yeah
21:14:02 mriedem so it blows here https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L741
21:14:27 mnaser yup, and the block device connection_info is set to null at this point
21:15:02 mnaser note if i quickly hit the db, i see it flap from NULL (actual SQL null, when a new record is created) to null (json-ified null, when the save happens, with a missing connection_info)
21:15:43 mriedem probably b/c of this https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L681
21:15:53 mriedem if connection_info_string != self._bdm_obj.connection_info:
21:15:56 mriedem if '' != None:
21:16:24 mnaser mriedem: more accurately if 'null' != None afaik
21:16:30 mriedem oh right yeah
21:16:42 mnaser hence i think this will fix it https://review.openstack.org/#/c/645352/
21:16:44 mriedem ok and then this blows up right? https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/utils.py#L1096
21:16:50 mriedem i mean, that is true,
21:16:54 mriedem then jsonutils.loads(bdm.connection_info).get('connector')
21:16:55 mriedem blows up
21:17:02 mnaser yeah

Earlier   Later