| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-03 | |||
| 14:37:13 | rouk | So... What's the fix? Is it broken? Am I using it wrong? | |
| 14:38:06 | sean-k-mooney | sorry im currently in the nova call. but ill try and take a look when its done | |
| 14:39:56 | rouk | Yeah sure thing, for now I have nova-api configured to use the nova db, instead of nova_cell0, which "fixes" the issue for now. But maybe that breaks other things I'm not aware of. It's just a dev location so I don't mind if I destroy the db. | |
| 14:41:02 | sean-k-mooney | that will only work for a singel cell deployment | |
| 14:41:33 | sean-k-mooney | you can deploy the metadata serivce seperatly and you can deploy a different metata api instance per cell | |
| 14:46:51 | rouk | Yeah, but in a single cell deployment should it be writing to nova then trying to read the data from nova_cell0? Looks like something has to be wrong. | |
| 14:47:17 | rouk | New data shows up in nova, API tries to read it from nova_cell0 and never sees it. | |
| 14:55:41 | sean-k-mooney | so the correct work around is to deploy a seperate metadta service and point it to the cell db | |
| 14:56:00 | sean-k-mooney | but keep the nova-api actully poining only at the api db | |
| 14:56:20 | sean-k-mooney | but we should look into this and try and fix it so that is not required | |
| 15:00:45 | rouk | Yeah. I'll look into deploying metadata separate, not sure how/if kolla has it supported yet, but I'll figure it out. Anything I can do to help fix the issue upstream? | |
| 15:04:15 | rouk | Also have i screwed my db by pointing directly at the nova db on nova-api and nova-scheduler? | |
| 15:04:35 | rouk | As that current workaround didn't have much thought put into the consequences | |
| 15:16:05 | sean-k-mooney | you likely have not broke them but you should not do that | |
| 15:16:46 | sean-k-mooney | dansmith: ^ the api db should not point directly to the cell db right. it looks like the block device mapping are only in the cell db so they are missing form the metadata responce | |
| 15:17:10 | sean-k-mooney | pointing nova-api at the cell db instead of cell0 seams to fix it but obvioly not the right thing to do | |
| 15:17:30 | sean-k-mooney | the workaround im suggesting is deploy a seperate metadta api and have it point to the cell db for now | |
| 15:18:13 | sean-k-mooney | rouk: i think we are just not correctly downcalling form the api to the cell db to get this info | |
| 16:24:35 | bauzas | beer'o time | |
| 16:27:55 | rouk | sean-k-mooney: seems like kolla doesnt have any tooling for deploying per-cell metadata yet, is there some documentation on whats actually needed to do it? any docs i can find from nova are super vague on how per-cell meta works. | |
| 16:34:59 | rouk | seems like the easiest fix would be to keep api connected to the cell's db until its fixed, unless theres some implementation docs i can use to add per-cell meta to kolla. | |
| 17:14:07 | lbragstad | gmann johnthetubaguy i see the policy stuff made it on the retrospective | |
| 17:14:34 | johnthetubaguy | lbragstad: yeah, just saying it was good it landed | |
| 17:14:35 | lbragstad | gmann johnthetubaguy i tried joining, but i was a late | |
| 17:14:52 | lbragstad | is there anything that we can take out of that discussion? | |
| 17:19:02 | gmann | lbragstad: few things we discussed during oslo-nova session on Monday | |
| 17:19:12 | gmann | on json format stuff | |
| 17:20:17 | gmann | lbragstad: L80 - https://etherpad.opendev.org/p/oslo-victoria-topics | |
| 17:21:14 | gmann | especially how to migrate json format to yaml in smooth way | |
| 17:21:19 | lbragstad | gmann ah - yeah | |
| 17:27:51 | sean-k-mooney | rouk: well you only have one cell so i would leave the main api pointing to the api db/cell 0 and have the seperate metadata service point to the singel cell | |
| 17:28:19 | sean-k-mooney | kolla does support a spereate metadtaa api i belive so that should be doable. i think | |
| 17:28:40 | rouk | yeah, but i dont know how to set up a separate metadata service, docs arent really specific on what to do, and id have to understand the process before i can add it to kolla. | |
| 17:28:56 | rouk | i dont see any tasks for it in kolla | |
| 17:31:14 | sean-k-mooney | ya just looked too it looks like they dont support it | |
| 17:32:08 | rouk | yeah, so im debating between adding support for it, which... theres little docs other than spec drawing for me to implement it, or just leaving nova-api on wrong db till i need multi-cell or the nova bug is fixed | |
| 17:32:44 | sean-k-mooney | rouk: so currently you are just chainging [database]/connection | |
| 17:32:51 | rouk | yes. | |
| 17:32:53 | sean-k-mooney | to point to your nova db which is cell 1 | |
| 17:33:06 | sean-k-mooney | and ytou are not changin [api_database]/connection | |
| 17:33:12 | rouk | correct. | |
| 17:33:34 | sean-k-mooney | im not sure if that would implcitly be harmful | |
| 17:34:09 | sean-k-mooney | rouk: the schelder shoudl not need to be modifed by the way | |
| 17:34:14 | sean-k-mooney | it would only be nova-api | |
| 17:34:45 | rouk | yeah, i need to revert that change, kolla puts them onto the same config, i need to make my change more specific and split it up a bit | |
| 17:35:06 | sean-k-mooney | ya that is simple to do with kolla | |
| 17:35:20 | rouk | yeah, just going to do a nova-api.conf override | |
| 17:35:52 | sean-k-mooney | yep | |
| 17:36:06 | sean-k-mooney | i really like there config override system | |
| 17:36:25 | sean-k-mooney | you know htat you can use jinja varables in your config overrides too | |
| 17:37:54 | sean-k-mooney | so you can put connection = mysql+pymysql://{{ nova_cell0_database_user }}:{{ nova_cell0_database_password }}@{{ nova_cell0_database_address }}/{{ nova_cell0_database_name }} | |
| 17:37:56 | sean-k-mooney | max_pool_size = 50 | |
| 17:38:00 | sean-k-mooney | in your config | |
| 17:38:16 | sean-k-mooney | and just change teh {{nova_cell0_database_name}} bit | |
| 17:38:47 | rouk | yep, already got the template change written when i overrode the .j2 file originally, just copying my change into nova-api instead | |
| 17:39:28 | sean-k-mooney | rouk: sure just making sure you know the {{}} syntax works most people dont realise that the overrides are processed as jinja templates too | |
| 17:41:48 | rouk | i hope i dont have to write nova-api-metadata myself for victoria or something, since yeah, ill probably need per-cell metadata soon, one site will be extending past 100 hypervisors eventually. metadata will probably be fiiiine till 200 hypervisors before i start having perf issues. | |
| 17:42:10 | rouk | how bad will the fix be on the nova side to fix single metadata? | |
| 17:42:37 | sean-k-mooney | rouk: you can run multiple instance of thte metadta service today with kolla | |
| 17:43:05 | sean-k-mooney | so you should be fine well beyond 200 instance provided you configure memcache to cache between the differen metadata services | |
| 17:43:34 | rouk | yeah, i have a bunch, its just going to suck when theyre spread over geographic locations, or large networks. im talking 200+ hypervisors, 100-200 instances each | |
| 17:43:52 | rouk | right now im at 80something hypervisors. | |
| 17:43:56 | sean-k-mooney | rouk: but regarding the fix sinc e i have not looked at why we are not geting the info form the cell db im not sure but i suspect it should not be supper invaisve to fix | |
| 17:44:23 | rouk | should i open a bug for tracking? | |
| 17:44:30 | sean-k-mooney | yes definetly | |
| 17:46:07 | rouk | alright, ill have that up once i have my kolla change deployed and verify my users are happy. | |
| 18:08:04 | josephillips | hey | |
| 18:08:21 | josephillips | someone with idea of how to solve this The instance sync for host 'Computenode' did not match. Re-created its InstanceList. | |
| 18:08:21 | openstack | Launchpad bug 1881944 in OpenStack Compute (nova) "nova-api returns empty block-device-mapping in metadata queries" [Undecided,New] | |
| 18:08:21 | rouk | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1881944 | |
| 18:08:27 | josephillips | happend eventualy | |
| 18:10:17 | josephillips | i had a issue with placement api that is already resolved | |
| 18:10:25 | josephillips | but this is the one error left on scheduler | |
| 18:20:41 | artom | mordred, hey, the functional tests in sdk - they hit a real devstack-deployed cloud in some of the zuul jobs in CI, right? | |
| 18:20:58 | artom | Or am I completely misunderstanding the code? | |
| 18:32:38 | mordred | artom: yup, that's right | |
| 18:33:08 | artom | mordred, cool, thanks. That was one of the points that came out of the discussion on the nova room this morning - how to test the SDK in CI | |
| 18:33:17 | artom | Good to know there's a framework already in place | |
| 18:33:31 | artom | Though I think no multinode testing, right? So live migration for instance isn't tested | |
| 18:33:33 | mordred | artom: all of the openstacksdk-functional-* jobs (and everything in openstack/tests/functional run against devstack | |
| 18:34:01 | mordred | artom: that's right - but I imagine if there is a multi-node devstack job that sets up multinode it would not be hard to add one | |
| 18:34:24 | artom | mordred, yeah, working on that now. | |
| 18:34:32 | artom | (If it seems disorganized, that's because it is) | |
| 18:34:42 | mordred | you're describing my life :) | |
| 18:34:53 | artom | (Expect a flurry of WIP patches as I organise my thoughts by working trough them) | |
| 18:35:05 | mordred | artom: looking forward to them | |
| 18:35:50 | artom | mordred, awesome, thanks | |
| 18:35:59 | mordred | artom: fwiw - we *always* run nova tests no matter what the devstack config is but we have a bunch of different devstack configs - so with neutron advanced services, with magnum, with senlin, with masakari etc | |
| 18:36:02 | artom | I gotta take the kiddos to the park first though :P | |
| 18:36:06 | mordred | but none of those configs don't have nova | |
| 18:36:39 | artom | mordred, that sounds reasonnable | |
| 18:39:42 | dansmith | artom: mordred: sounds like that's what we're looking for, and important to note that (apparently) sdk functional is more like nova integrated | |
| 18:51:33 | mordred | dansmith: english is hard | |
| 19:23:31 | alex_xu | sean-k-mooney: gibi sorry, just saw yulong's message, I tried chrome, but I think it is network issue. With intel proxy doesn't work, without proxy even can't open the page. but anyway we already switch to zoom :) | |
| 21:39:10 | CeeMac | Evening | |
| 21:42:49 | CeeMac | I'm trying to confirm some details on what exactly 'active' means in context to (I believe) how nova-usage reports cpu-hours, gb-hours etc to horizon for Admin | Overview | Usage Summary. My original understanding was that 'Active' equated to time instances spent in 'running' state it appears that may not be the case. Is anyone able to confirm as what documentation I could find wasn't clear. | |
| 21:43:39 | CeeMac | When looking at usage over a period e.g for the previous month. | |
| 21:57:42 | CeeMac | Digging in to the api it looks like 'active' may relate to 'server_usages.state' but I can't find a list of alternative values to give that context. For example is an instance 'active' when it is shutdown, does it need to be shelved or deleted to change from 'active' state and not accumulate usage values? | |
| 22:15:34 | sean-k-mooney | artom: if you need a multi node devstack just change the nodepool nodeset | |
| 22:16:02 | sean-k-mooney | the devstack ansiable playbooks support muiltnode so it shoudl be easy to make the jobs multinode | |