| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 17:10:34 | prometheanfire | sean-k-mooney: ok, thought the nova-compute service handled registering itself | |
| 17:10:49 | mriedem | prometheanfire: it does | |
| 17:10:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556 | |
| 17:10:55 | mriedem | SamYaple: which release are you on? | |
| 17:11:17 | SamYaple | mriedem: ocata/pike/queens across a few environments | |
| 17:11:27 | sean-k-mooney | SamYaple:ya perhapos never tried it. with all the forign key on the table however it might be more hassel then its worth | |
| 17:11:38 | mriedem | ok, because we didn't until recently cleanup after ourselves wrt placement when deleting a nova-compute service record | |
| 17:11:49 | mriedem | thta's fixed now, but you'd have to make sure you have it, otherwise you have to cleanup placement entries yourself | |
| 17:12:02 | sean-k-mooney | prometheanfire: if you have the right setting in your conf the nova-compute agent will take care of populating the placement api for you | |
| 17:12:20 | SamYaple | ah, yes. this process has only been done on the older ocata clusters. We should be queens everywhere by end of year though. | |
| 17:12:28 | prometheanfire | sean-k-mooney: nova-status upgrade check still shows the warning | |
| 17:12:31 | SamYaple | mriedem: my sql are me cleaning up the placement entries :) | |
| 17:12:43 | mriedem | fyi https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80 | |
| 17:13:49 | SamYaple | mriedem: oh b-e-a-utiful. Super helpful. gonna do an internal backport on that for our ocata stuff. got a few hundred more nodes to go | |
| 17:13:53 | sean-k-mooney | prometheanfire: for devstack the plamcenet section looks like this http://paste.openstack.org/show/729431/ | |
| 17:14:28 | prometheanfire | sean-k-mooney: yep, looks right (looks like mine) | |
| 17:15:22 | sean-k-mooney | prometheanfire: strange and openstack resource provider list is does not have the compute nodes? | |
| 17:15:46 | prometheanfire | sean-k-mooney: http://paste.openstack.org/show/729432/ | |
| 17:16:04 | mriedem | SamYaple: well, be warned, it's a hefty series of backports | |
| 17:16:28 | mriedem | this is just pike https://review.openstack.org/#/q/topic:bug/1756179+(status:open+OR+status:merged)+branch:stable/pike | |
| 17:16:30 | prometheanfire | sean-k-mooney: openstack service provider list | |
| 17:16:32 | prometheanfire | that? | |
| 17:16:49 | sean-k-mooney | no "openstack resource provider list" | |
| 17:17:05 | sean-k-mooney | if you do not have that you are missing the placement osc plugin | |
| 17:17:16 | prometheanfire | ok | |
| 17:17:21 | mriedem | prometheanfire: are there errors in the nova-compute logs? | |
| 17:17:44 | prometheanfire | mriedem: which errors you looking for? | |
| 17:17:44 | sean-k-mooney | prometheanfire: https://github.com/openstack/osc-placement for future reference. it should be on pypi too | |
| 17:17:52 | mriedem | prometheanfire: umm, any errors? | |
| 17:18:14 | mriedem | presumably "failed to create resource provider in placement" or something like that | |
| 17:18:16 | prometheanfire | mriedem: none now, I rebuilt all my systems from the ground up to fix the odd python issue | |
| 17:18:23 | SamYaple | mriedem: sorry, i said backport. i mean hack something in based on that commit. i stay pretty close to upstream | |
| 17:18:40 | prometheanfire | mriedem: testing now via ' su - nova -c "/usr/lib/python-exec/python3.5/nova-compute --config-file /etc/nova/nova-compute.conf --config-file /etc/nova/nova.conf" ' | |
| 17:19:03 | mriedem | SamYaple: ok, well, good luck. :) if that series of backports on pike also applies to ocata (i'm not sure if they all do), we could also backport them to ocata upstream | |
| 17:19:07 | mriedem | all official like | |
| 17:20:53 | sean-k-mooney | prometheanfire: you should get somting like "ResourceProviderCreationFailed" if the placement info is missing/wrong in the conf | |
| 17:20:58 | prometheanfire | mriedem: http://paste.openstack.org/show/729433/ | |
| 17:21:00 | prometheanfire | lgtm | |
| 17:21:19 | melwitt | . | |
| 17:21:20 | sean-k-mooney | prometheanfire: that what i get when i comment it out but the agent keeps running which i was surrprised by | |
| 17:22:55 | openstackgerrit | Merged openstack/nova master: Removing pip-missing-reqs from default tox jobs https://review.openstack.org/599442 | |
| 17:23:10 | mriedem | prometheanfire: it's likely logged at debug | |
| 17:23:18 | prometheanfire | ah, k | |
| 17:23:23 | mriedem | do you see the resource provider in placement now? | |
| 17:23:48 | prometheanfire | mriedem: I'm packaging osc-placement now | |
| 17:24:00 | SamYaple | mriedem: ack. will do | |
| 17:24:04 | sean-k-mooney | mriedem: no its logging at error if it fails | |
| 17:26:51 | prometheanfire | sean-k-mooney: mriedem does osc-placement need to be installed on the compute host? | |
| 17:27:28 | cdent | prometheanfire: no, it's only an openstack client plugin, for humans | |
| 17:27:50 | prometheanfire | that's what I thought | |
| 17:32:53 | sean-k-mooney | prometheanfire: its not needed but it makes debuging placment issues simpeler as you dont have to dive into the db | |
| 17:34:04 | prometheanfire | openstack resource provider list - shows my two nodes | |
| 17:34:38 | sean-k-mooney | prometheanfire: then placement is happy with them. what was the warning you were getting with nova status-check? | |
| 17:35:29 | prometheanfire | one is generation 76 and one is generation 34 though | |
| 17:35:59 | prometheanfire | sean-k-mooney: http://paste.openstack.org/show/729434/ | |
| 17:36:59 | sean-k-mooney | cdent: do you no if nova-status upgrade check uses the api or try to connect to the db directly? | |
| 17:37:23 | cdent | sean-k-mooney: api. it talks to the db for nova stuff, api for placement stuff | |
| 17:37:54 | cdent | sean-k-mooney: yes, it is useful for debugging, but it doesn't have to be _on_ the compute node... | |
| 17:38:32 | sean-k-mooney | prometheanfire: cdent yes. im wondering if nova-status if not finding the placement endoint or somthing in prometheanfire case | |
| 17:38:56 | sean-k-mooney | cdent: sorry ^ was ment for you | |
| 17:39:43 | sean-k-mooney | prometheanfire: do you have the placement setting in the nova.conf also or jsut /etc/nova/nova-cpu.conf? | |
| 17:40:14 | prometheanfire | sean-k-mooney: I have it in nova.conf for all hosts (nova-api and compute hosts) | |
| 17:41:00 | prometheanfire | online-data migrations fail too, I think it's looking in the wrong db here | |
| 17:41:28 | sean-k-mooney | ok i was going to suggest using --config-file=/path/to/conf to make sure its not reading values form another location | |
| 17:41:44 | cdent | sorry sean-k-mooney, I haven't been tracking prometheanfire's situation, just jumping at random points. in too many conversations at once | |
| 17:42:08 | sean-k-mooney | cdent: no worries. | |
| 17:42:54 | sean-k-mooney | prometheanfire: i would try "nova-status --debug --config-dir /etc/nova/nova.conf upgrade check" | |
| 17:43:19 | prometheanfire | same | |
| 17:44:22 | prometheanfire | sean-k-mooney: does placement look at the cell_mappings table of the api db? | |
| 17:45:05 | sean-k-mooney | prometheanfire: i dont think placement does but if you have not do the cellsv2 discover host thing this might have issues | |
| 17:46:58 | sean-k-mooney | prometheanfire: actull i think the output is missleading | |
| 17:47:24 | sean-k-mooney | prometheanfire: i get http://paste.openstack.org/show/729436/ | |
| 17:48:05 | prometheanfire | sean-k-mooney: that's diferent than mine | |
| 17:49:04 | prometheanfire | nova-manage cell_v2 list_hosts | |
| 17:49:10 | prometheanfire | shows my hosts as not mapped to a cell | |
| 17:49:13 | sean-k-mooney | yes it is but 1 i got a success and its an all in one devstrck that is passing tempest test so it works | |
| 17:49:28 | sean-k-mooney | so i think the test might be wrong | |
| 17:51:07 | sean-k-mooney | hum ok well ill try to look into this more tomorow. my brain is fried for today so im going to call it a day o/ | |
| 17:52:33 | prometheanfire | k | |
| 17:57:15 | prometheanfire | list cells backtraces | |
| 17:57:17 | prometheanfire | http://paste.openstack.org/show/729437/ | |
| 17:57:26 | mriedem | prometheanfire: you still don't have a resource provider in placement but no errors in the n-cpu logs? | |
| 17:58:59 | prometheanfire | mriedem: shows my two nodes | |
| 17:59:29 | mriedem | huh your cells must not have names | |
| 17:59:34 | prometheanfire | my api_db has two entries for cell_mappings | |
| 17:59:36 | prometheanfire | select * from cell_mappings; | |
| 17:59:38 | mriedem | which is the TypeError | |
| 17:59:51 | prometheanfire | ya, one of them has no name | |
| 17:59:59 | mriedem | ok that's a bug clearly | |
| 18:00:07 | mriedem | you want to report it or shall i? | |
| 18:01:29 | prometheanfire | I'm still figuring stuff out | |
| 18:01:53 | prometheanfire | I think a lot of my bugs in this area are because I moved to cells before it was ready | |
| 18:02:39 | prometheanfire | list_hosts now has the cell name :D | |
| 18:02:48 | prometheanfire | mriedem: can I delete cell0 or is that still used? | |
| 18:03:33 | mriedem | cell0 is required | |
| 18:03:44 | mriedem | it's where instances records are created for things that fail to schedule | |
| 18:03:55 | prometheanfire | k | |
| 18:04:00 | mriedem | "for cell in sorted(cell_mappings, key=lambda _cell: _cell.name):" | |
| 18:04:02 | mriedem | oops | |
| 18:04:07 | mriedem | "because I moved to cells before it was ready" | |