| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-21 | |||
| 20:42:36 | sean-k-mooney | pip just looks ate the requiremets.txt files | |
| 20:42:41 | efried | sure, but I would expect the l-c job to know about them. | |
| 20:42:56 | efried | ...about the lower constraints for the projects that have lower constraints. | |
| 20:43:10 | sean-k-mooney | the lower constratin job does pip install -r requriements.txt -c lower-constratints.txt | |
| 20:43:40 | sean-k-mooney | only one constratint file. nova is passed when installing the deps | |
| 20:43:51 | efried | mm, okay, then yeah, I guess it should only be observing my own l-c huh | |
| 20:46:14 | sean-k-mooney | for what its worth the job does look broken | |
| 20:46:21 | sean-k-mooney | http://logs.openstack.org/46/621646/20/check/openstack-tox-lower-constraints/8b49dc9/tox/lower-constraints-1.log | |
| 20:46:59 | sean-k-mooney | we are passing bot an upper-constratins.txt and lower-constraints.txt to pip | |
| 20:48:30 | sean-k-mooney | ok i see what going on | |
| 20:48:39 | sean-k-mooney | ill see if i can fix that | |
| 20:49:03 | sean-k-mooney | our install command include upper-constratints | |
| 20:51:24 | efried | sean-k-mooney: you mean L18 of tox.ini? | |
| 20:51:43 | sean-k-mooney | yes | |
| 20:52:09 | sean-k-mooney | im overriding hte install_command local to not do that in the lower-constraints env and ill see what hapens | |
| 20:52:20 | sean-k-mooney | i can see it endup using 2.7.7 in the gate | |
| 20:52:31 | sean-k-mooney | it should have been using 2.6.8 or whatever | |
| 20:53:13 | sean-k-mooney | well that failed instantly | |
| 20:53:16 | efried | so there's nothing tox-magical about the "lower-constraints" name like there is for pyXX that would be smart enough to use a fresh install_command? | |
| 20:53:41 | efried | failed instantly because a l-c is too low? | |
| 20:53:42 | sean-k-mooney | no lower-constraints is an openstack thing | |
| 20:53:49 | efried | okay. | |
| 20:54:06 | sean-k-mooney | it faild on the postgressql issue | |
| 20:54:10 | efried | so we've been borked for a while and real l-cs have been creeping past us. | |
| 20:54:29 | efried | edmondsw: Thanks for the find there! | |
| 20:54:33 | sean-k-mooney | ya for basicly ever i think looking at the git blame | |
| 20:54:50 | sean-k-mooney | the install command predate adding lower constratints i think | |
| 20:54:50 | efried | well, we've done quite a bit to tox.ini in the last release or two. | |
| 20:55:01 | efried | ...but not sure if those have been touched. | |
| 20:55:19 | sean-k-mooney | ill bump the lower constatin to 2.7 and push a patch shortly | |
| 20:55:30 | efried | sean-k-mooney: Well, that might not be the only transgressor | |
| 20:55:35 | efried | but I guess you'll find out :) | |
| 20:55:51 | sean-k-mooney | i was trying not to say that out loud but yes | |
| 20:57:20 | sean-k-mooney | i should proably file a bug for this. ya the next issue is the zvm sdk | |
| 20:57:57 | sean-k-mooney | i think i fixed this error actully it complingin about the python version support | |
| 20:58:45 | sean-k-mooney | i shoudl proably file a bug for this. shoudl this be a RC2 thing | |
| 21:00:57 | openstackgerrit | Merged openstack/nova-specs master: Re-propose count quota usage from placement https://review.openstack.org/645302 | |
| 21:04:36 | mnaser | im struggling to trace the path of failure but i have a fix for this :\ | |
| 21:07:02 | openstackgerrit | Mohammed Naser proposed openstack/nova master: wip: bdm: store empty object as connection_info by default https://review.openstack.org/645352 | |
| 21:07:27 | mnaser | if someone has a few cycles to spare, i'm going to keep digging but i don't know the code that well to know why it's being saved with 'null' | |
| 21:08:23 | sean-k-mooney | mnaser: if the value is ever set to None it will be serialised to 'null' | |
| 21:08:55 | sean-k-mooney | i can try and take a look at it in a bit but i also dont know the bdm code that well | |
| 21:08:56 | mnaser | sean-k-mooney: yeah.. but i am trying to figure out why it would either be a) set to None or b) .get('connection_info') fails and returns None | |
| 21:09:30 | mnaser | oh interesting | |
| 21:09:34 | sean-k-mooney | if its trying to delete it before the block device mappings are constructed then that might be the issue | |
| 21:10:03 | mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L284-L288 | |
| 21:10:14 | mnaser | by default, the connection_info field defaults to 'NULL' | |
| 21:10:23 | mnaser | so that means self['connection_info'] = None .. | |
| 21:10:40 | mriedem | isn't the instance.host set before that though? | |
| 21:10:46 | mriedem | the instance_claim sets the instance.host | |
| 21:10:47 | sean-k-mooney | ya which just porpagates that default | |
| 21:10:51 | mriedem | so you shouldn't get a local delete flow in the API | |
| 21:11:09 | mnaser | mriedem: this is a case where instance.host = None so i think what happens is that the build aborts a few times on the compute host | |
| 21:11:17 | mnaser | and then when it fails to build it, instance.host goes to None | |
| 21:11:31 | mnaser | because cinder blows up when it tries to create a volume and you're out of quota | |
| 21:11:57 | mnaser | this isnt a case of instance.host = None because we're in cell0, it's because we failed to build the instance N times and gave up | |
| 21:12:12 | mriedem | anything failing in _prep_block_device automatically aborts the build | |
| 21:12:14 | mriedem | there is no reschedule | |
| 21:12:38 | mriedem | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L2317 | |
| 21:12:43 | mriedem | especially if you hit overquota | |
| 21:12:52 | mnaser | yeah im def hitting overquota because of the error logs | |
| 21:13:01 | mnaser | "Failed to create block device for instance due to exceeding volume related resource quota. Error: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 80G, quota is 6144G and 6080G has been consumed.: OverQuota: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 80G, quota is 6144G and 6080G has been consumed." | |
| 21:13:23 | mriedem | that should reset the instance.host to None here https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L2018 | |
| 21:13:23 | mnaser | so that code is getting exercised | |
| 21:13:45 | mriedem | this is booting a volume from an image yeah? | |
| 21:13:46 | mnaser | which means a follow up delete will result in a local delete path? | |
| 21:13:47 | mnaser | yeah | |
| 21:14:02 | mriedem | so it blows here https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L741 | |
| 21:14:27 | mnaser | yup, and the block device connection_info is set to null at this point | |
| 21:15:02 | mnaser | note if i quickly hit the db, i see it flap from NULL (actual SQL null, when a new record is created) to null (json-ified null, when the save happens, with a missing connection_info) | |
| 21:15:43 | mriedem | probably b/c of this https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L681 | |
| 21:15:53 | mriedem | if connection_info_string != self._bdm_obj.connection_info: | |
| 21:15:56 | mriedem | if '' != None: | |
| 21:16:24 | mnaser | mriedem: more accurately if 'null' != None afaik | |
| 21:16:30 | mriedem | oh right yeah | |
| 21:16:42 | mnaser | hence i think this will fix it https://review.openstack.org/#/c/645352/ | |
| 21:16:44 | mriedem | ok and then this blows up right? https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/utils.py#L1096 | |
| 21:16:50 | mriedem | i mean, that is true, | |
| 21:16:54 | mriedem | then jsonutils.loads(bdm.connection_info).get('connector') | |
| 21:16:55 | mriedem | blows up | |
| 21:17:02 | mnaser | yeah | |
| 21:17:40 | mnaser | that's as far as i got, but i don't know what triggers a save() which pushes the json stuff in the db in the failing path | |
| 21:18:14 | mriedem | that code is rife with tricky decorators that save changes on methods | |
| 21:18:16 | mriedem | so could be one of those | |
| 21:18:20 | mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L645 | |
| 21:18:27 | mnaser | so if `refresh_connection_info` is called at any given moment | |
| 21:18:37 | mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L648-L650 | |
| 21:18:40 | mnaser | will return | |
| 21:18:44 | mnaser | and then it will call the decorator to save | |
| 21:19:09 | mnaser | self['connection_info'] by default is None | |
| 21:19:24 | mriedem | i don't think refresh_connection_info would be getting called during spawn | |
| 21:19:40 | mnaser | the only place i noticed calls it is refresh_conn_infos | |
| 21:19:43 | mriedem | but yeah that update_db decorator is what could be doing it | |
| 21:20:18 | sean-k-mooney | efried: .. we pin our lower constating for warlock to one explcitly blacklisted by python-glance client... | |
| 21:20:22 | mnaser | ah | |
| 21:20:23 | sean-k-mooney | the fun continues | |
| 21:20:31 | mriedem | mnaser: i think that's only called here https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L1779 | |
| 21:20:33 | mriedem | on like move ops and such | |
| 21:20:44 | mnaser | indeed thats the only reference to it that i find | |
| 21:21:23 | efried | sean-k-mooney: Presumably it doesn't hurt to elevate the warlock l-c to... whatever glanclient is using | |