| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 16:56:43 | prometheanfire | sean-k-mooney: btw, did you see my comment about the database connection string in the apidb not being urlencoded? | |
| 16:56:46 | openstackgerrit | Merged openstack/nova master: Combine error handling blocks in _do_build_and_run_instance https://review.openstack.org/545960 | |
| 16:57:09 | prometheanfire | too far in backlog now | |
| 16:58:21 | prometheanfire | update cell_mappings set database_connection='URLENCODED_CONNECTION_STRING' where uuid='0000' | |
| 17:01:05 | SamYaple | what actions, if any, can a user do that will trigger an 'instance.update' notification? having troble figuring it out from the code | |
| 17:03:24 | prometheanfire | There are no compute resource providers in the Placement service but there are 2 compute nodes in the deployment. This means no compute nodes are reporting into the Placement service and need to be upgraded and/or fixed. | |
| 17:03:40 | prometheanfire | but the compute nodes have the placement info in their config, not sure what else to do there | |
| 17:05:27 | mriedem | SamYaple: do you have notify_on_state_change set? | |
| 17:05:34 | SamYaple | mriedem: yes | |
| 17:05:43 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notify_on_state_change | |
| 17:05:44 | mriedem | to what? | |
| 17:05:57 | SamYaple | prometheanfire: just checked scrollback, i just dealt with that issue! on a few nodes. It was caused by reprovisioning my compute nodes with teh same names but migrating the instances off with evacuate. when it came back the database was messed up | |
| 17:06:46 | mriedem | prometheanfire: are you on pike or master/ | |
| 17:06:47 | mriedem | ? | |
| 17:06:50 | prometheanfire | SamYaple: oh? | |
| 17:06:50 | sean-k-mooney | prometheanfire: jaypipes or cdent might be able to help with that. im guessiog your missing the placement client and or config section in the nova config but honestly that is jsut a guess | |
| 17:06:55 | prometheanfire | mriedem: rocky | |
| 17:07:42 | SamYaple | mriedem: so it *only* sends updates with state changes? if i read that correctly? (sorry, chasing down a bug in an inherited notifications reader with bad comments, so im trying to be explicit) | |
| 17:08:03 | SamYaple | prometheanfire: yes, it had to do with the old instances that had been in an ERROR state before the node went down | |
| 17:08:05 | mriedem | prometheanfire: i relatively recently talked with dansmith about trying to parse/encode/decode the db connection string in the cell_mappings table and i think the consensus was if you have special stuff in the url, you need to encode it beforehand | |
| 17:08:05 | prometheanfire | sean-k-mooney: there's a separate placement client? as in pypi type thing? | |
| 17:08:12 | sean-k-mooney | SamYaple: that kind of makes sense. since you used the same host name the compute agent will get teh compute servie record for the previous install | |
| 17:08:14 | SamYaple | prometheanfire: ill try to pull my sql queries to fix it | |
| 17:08:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556 | |
| 17:08:30 | prometheanfire | mriedem: ah, somehow I got it in the db | |
| 17:09:51 | sean-k-mooney | prometheanfire: there is a placement osc plugin but when i said placement client i was referint to the devstack placement clinet service but honestly dont know what that does | |
| 17:09:58 | mriedem | SamYaple: it looks like a hack way to trigger the instance.update notififcation is to change metadata on the instance | |
| 17:10:00 | SamYaple | sean-k-mooney: would a possible workaround for my use case possibly be solved simply by deleting the service record between reprovisioning? | |
| 17:10:32 | SamYaple | mriedem: ah! perfect! ok. so thats the hack this code is trying to tap into. that aligns on my end. thank you | |
| 17:10:34 | prometheanfire | sean-k-mooney: ok, thought the nova-compute service handled registering itself | |
| 17:10:49 | mriedem | prometheanfire: it does | |
| 17:10:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556 | |
| 17:10:55 | mriedem | SamYaple: which release are you on? | |
| 17:11:17 | SamYaple | mriedem: ocata/pike/queens across a few environments | |
| 17:11:27 | sean-k-mooney | SamYaple:ya perhapos never tried it. with all the forign key on the table however it might be more hassel then its worth | |
| 17:11:38 | mriedem | ok, because we didn't until recently cleanup after ourselves wrt placement when deleting a nova-compute service record | |
| 17:11:49 | mriedem | thta's fixed now, but you'd have to make sure you have it, otherwise you have to cleanup placement entries yourself | |
| 17:12:02 | sean-k-mooney | prometheanfire: if you have the right setting in your conf the nova-compute agent will take care of populating the placement api for you | |
| 17:12:20 | SamYaple | ah, yes. this process has only been done on the older ocata clusters. We should be queens everywhere by end of year though. | |
| 17:12:28 | prometheanfire | sean-k-mooney: nova-status upgrade check still shows the warning | |
| 17:12:31 | SamYaple | mriedem: my sql are me cleaning up the placement entries :) | |
| 17:12:43 | mriedem | fyi https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80 | |
| 17:13:49 | SamYaple | mriedem: oh b-e-a-utiful. Super helpful. gonna do an internal backport on that for our ocata stuff. got a few hundred more nodes to go | |
| 17:13:53 | sean-k-mooney | prometheanfire: for devstack the plamcenet section looks like this http://paste.openstack.org/show/729431/ | |
| 17:14:28 | prometheanfire | sean-k-mooney: yep, looks right (looks like mine) | |
| 17:15:22 | sean-k-mooney | prometheanfire: strange and openstack resource provider list is does not have the compute nodes? | |
| 17:15:46 | prometheanfire | sean-k-mooney: http://paste.openstack.org/show/729432/ | |
| 17:16:04 | mriedem | SamYaple: well, be warned, it's a hefty series of backports | |
| 17:16:28 | mriedem | this is just pike https://review.openstack.org/#/q/topic:bug/1756179+(status:open+OR+status:merged)+branch:stable/pike | |
| 17:16:30 | prometheanfire | sean-k-mooney: openstack service provider list | |
| 17:16:32 | prometheanfire | that? | |
| 17:16:49 | sean-k-mooney | no "openstack resource provider list" | |
| 17:17:05 | sean-k-mooney | if you do not have that you are missing the placement osc plugin | |
| 17:17:16 | prometheanfire | ok | |
| 17:17:21 | mriedem | prometheanfire: are there errors in the nova-compute logs? | |
| 17:17:44 | sean-k-mooney | prometheanfire: https://github.com/openstack/osc-placement for future reference. it should be on pypi too | |
| 17:17:44 | prometheanfire | mriedem: which errors you looking for? | |
| 17:17:52 | mriedem | prometheanfire: umm, any errors? | |
| 17:18:14 | mriedem | presumably "failed to create resource provider in placement" or something like that | |
| 17:18:16 | prometheanfire | mriedem: none now, I rebuilt all my systems from the ground up to fix the odd python issue | |
| 17:18:23 | SamYaple | mriedem: sorry, i said backport. i mean hack something in based on that commit. i stay pretty close to upstream | |
| 17:18:40 | prometheanfire | mriedem: testing now via ' su - nova -c "/usr/lib/python-exec/python3.5/nova-compute --config-file /etc/nova/nova-compute.conf --config-file /etc/nova/nova.conf" ' | |
| 17:19:03 | mriedem | SamYaple: ok, well, good luck. :) if that series of backports on pike also applies to ocata (i'm not sure if they all do), we could also backport them to ocata upstream | |
| 17:19:07 | mriedem | all official like | |
| 17:20:53 | sean-k-mooney | prometheanfire: you should get somting like "ResourceProviderCreationFailed" if the placement info is missing/wrong in the conf | |
| 17:20:58 | prometheanfire | mriedem: http://paste.openstack.org/show/729433/ | |
| 17:21:00 | prometheanfire | lgtm | |
| 17:21:19 | melwitt | . | |
| 17:21:20 | sean-k-mooney | prometheanfire: that what i get when i comment it out but the agent keeps running which i was surrprised by | |
| 17:22:55 | openstackgerrit | Merged openstack/nova master: Removing pip-missing-reqs from default tox jobs https://review.openstack.org/599442 | |
| 17:23:10 | mriedem | prometheanfire: it's likely logged at debug | |
| 17:23:18 | prometheanfire | ah, k | |
| 17:23:23 | mriedem | do you see the resource provider in placement now? | |
| 17:23:48 | prometheanfire | mriedem: I'm packaging osc-placement now | |
| 17:24:00 | SamYaple | mriedem: ack. will do | |
| 17:24:04 | sean-k-mooney | mriedem: no its logging at error if it fails | |
| 17:26:51 | prometheanfire | sean-k-mooney: mriedem does osc-placement need to be installed on the compute host? | |
| 17:27:28 | cdent | prometheanfire: no, it's only an openstack client plugin, for humans | |
| 17:27:50 | prometheanfire | that's what I thought | |
| 17:32:53 | sean-k-mooney | prometheanfire: its not needed but it makes debuging placment issues simpeler as you dont have to dive into the db | |
| 17:34:04 | prometheanfire | openstack resource provider list - shows my two nodes | |
| 17:34:38 | sean-k-mooney | prometheanfire: then placement is happy with them. what was the warning you were getting with nova status-check? | |
| 17:35:29 | prometheanfire | one is generation 76 and one is generation 34 though | |
| 17:35:59 | prometheanfire | sean-k-mooney: http://paste.openstack.org/show/729434/ | |
| 17:36:59 | sean-k-mooney | cdent: do you no if nova-status upgrade check uses the api or try to connect to the db directly? | |
| 17:37:23 | cdent | sean-k-mooney: api. it talks to the db for nova stuff, api for placement stuff | |
| 17:37:54 | cdent | sean-k-mooney: yes, it is useful for debugging, but it doesn't have to be _on_ the compute node... | |
| 17:38:32 | sean-k-mooney | prometheanfire: cdent yes. im wondering if nova-status if not finding the placement endoint or somthing in prometheanfire case | |
| 17:38:56 | sean-k-mooney | cdent: sorry ^ was ment for you | |
| 17:39:43 | sean-k-mooney | prometheanfire: do you have the placement setting in the nova.conf also or jsut /etc/nova/nova-cpu.conf? | |
| 17:40:14 | prometheanfire | sean-k-mooney: I have it in nova.conf for all hosts (nova-api and compute hosts) | |
| 17:41:00 | prometheanfire | online-data migrations fail too, I think it's looking in the wrong db here | |
| 17:41:28 | sean-k-mooney | ok i was going to suggest using --config-file=/path/to/conf to make sure its not reading values form another location | |
| 17:41:44 | cdent | sorry sean-k-mooney, I haven't been tracking prometheanfire's situation, just jumping at random points. in too many conversations at once | |
| 17:42:08 | sean-k-mooney | cdent: no worries. | |
| 17:42:54 | sean-k-mooney | prometheanfire: i would try "nova-status --debug --config-dir /etc/nova/nova.conf upgrade check" | |
| 17:43:19 | prometheanfire | same | |
| 17:44:22 | prometheanfire | sean-k-mooney: does placement look at the cell_mappings table of the api db? | |
| 17:45:05 | sean-k-mooney | prometheanfire: i dont think placement does but if you have not do the cellsv2 discover host thing this might have issues | |
| 17:46:58 | sean-k-mooney | prometheanfire: actull i think the output is missleading | |