Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-06-03
14:47:17 rouk New data shows up in nova, API tries to read it from nova_cell0 and never sees it.
14:55:41 sean-k-mooney so the correct work around is to deploy a seperate metadta service and point it to the cell db
14:56:00 sean-k-mooney but keep the nova-api actully poining only at the api db
14:56:20 sean-k-mooney but we should look into this and try and fix it so that is not required
15:00:45 rouk Yeah. I'll look into deploying metadata separate, not sure how/if kolla has it supported yet, but I'll figure it out. Anything I can do to help fix the issue upstream?
15:04:15 rouk Also have i screwed my db by pointing directly at the nova db on nova-api and nova-scheduler?
15:04:35 rouk As that current workaround didn't have much thought put into the consequences
15:16:05 sean-k-mooney you likely have not broke them but you should not do that
15:16:46 sean-k-mooney dansmith: ^ the api db should not point directly to the cell db right. it looks like the block device mapping are only in the cell db so they are missing form the metadata responce
15:17:10 sean-k-mooney pointing nova-api at the cell db instead of cell0 seams to fix it but obvioly not the right thing to do
15:17:30 sean-k-mooney the workaround im suggesting is deploy a seperate metadta api and have it point to the cell db for now
15:18:13 sean-k-mooney rouk: i think we are just not correctly downcalling form the api to the cell db to get this info
16:24:35 bauzas beer'o time
16:27:55 rouk sean-k-mooney: seems like kolla doesnt have any tooling for deploying per-cell metadata yet, is there some documentation on whats actually needed to do it? any docs i can find from nova are super vague on how per-cell meta works.
16:34:59 rouk seems like the easiest fix would be to keep api connected to the cell's db until its fixed, unless theres some implementation docs i can use to add per-cell meta to kolla.
17:14:07 lbragstad gmann johnthetubaguy i see the policy stuff made it on the retrospective
17:14:34 johnthetubaguy lbragstad: yeah, just saying it was good it landed
17:14:35 lbragstad gmann johnthetubaguy i tried joining, but i was a late
17:14:52 lbragstad is there anything that we can take out of that discussion?
17:19:02 gmann lbragstad: few things we discussed during oslo-nova session on Monday
17:19:12 gmann on json format stuff
17:20:17 gmann lbragstad: L80 - https://etherpad.opendev.org/p/oslo-victoria-topics
17:21:14 gmann especially how to migrate json format to yaml in smooth way
17:21:19 lbragstad gmann ah - yeah
17:27:51 sean-k-mooney rouk: well you only have one cell so i would leave the main api pointing to the api db/cell 0 and have the seperate metadata service point to the singel cell
17:28:19 sean-k-mooney kolla does support a spereate metadtaa api i belive so that should be doable. i think
17:28:40 rouk yeah, but i dont know how to set up a separate metadata service, docs arent really specific on what to do, and id have to understand the process before i can add it to kolla.
17:28:56 rouk i dont see any tasks for it in kolla
17:31:14 sean-k-mooney ya just looked too it looks like they dont support it
17:32:08 rouk yeah, so im debating between adding support for it, which... theres little docs other than spec drawing for me to implement it, or just leaving nova-api on wrong db till i need multi-cell or the nova bug is fixed
17:32:44 sean-k-mooney rouk: so currently you are just chainging [database]/connection
17:32:51 rouk yes.
17:32:53 sean-k-mooney to point to your nova db which is cell 1
17:33:06 sean-k-mooney and ytou are not changin [api_database]/connection
17:33:12 rouk correct.
17:33:34 sean-k-mooney im not sure if that would implcitly be harmful
17:34:09 sean-k-mooney rouk: the schelder shoudl not need to be modifed by the way
17:34:14 sean-k-mooney it would only be nova-api
17:34:45 rouk yeah, i need to revert that change, kolla puts them onto the same config, i need to make my change more specific and split it up a bit
17:35:06 sean-k-mooney ya that is simple to do with kolla
17:35:20 rouk yeah, just going to do a nova-api.conf override
17:35:52 sean-k-mooney yep
17:36:06 sean-k-mooney i really like there config override system
17:36:25 sean-k-mooney you know htat you can use jinja varables in your config overrides too
17:37:54 sean-k-mooney so you can put connection = mysql+pymysql://{{ nova_cell0_database_user }}:{{ nova_cell0_database_password }}@{{ nova_cell0_database_address }}/{{ nova_cell0_database_name }}
17:37:56 sean-k-mooney max_pool_size = 50
17:38:00 sean-k-mooney in your config
17:38:16 sean-k-mooney and just change teh {{nova_cell0_database_name}} bit
17:38:47 rouk yep, already got the template change written when i overrode the .j2 file originally, just copying my change into nova-api instead
17:39:28 sean-k-mooney rouk: sure just making sure you know the {{}} syntax works most people dont realise that the overrides are processed as jinja templates too
17:41:48 rouk i hope i dont have to write nova-api-metadata myself for victoria or something, since yeah, ill probably need per-cell metadata soon, one site will be extending past 100 hypervisors eventually. metadata will probably be fiiiine till 200 hypervisors before i start having perf issues.
17:42:10 rouk how bad will the fix be on the nova side to fix single metadata?
17:42:37 sean-k-mooney rouk: you can run multiple instance of thte metadta service today with kolla
17:43:05 sean-k-mooney so you should be fine well beyond 200 instance provided you configure memcache to cache between the differen metadata services
17:43:34 rouk yeah, i have a bunch, its just going to suck when theyre spread over geographic locations, or large networks. im talking 200+ hypervisors, 100-200 instances each
17:43:52 rouk right now im at 80something hypervisors.
17:43:56 sean-k-mooney rouk: but regarding the fix sinc e i have not looked at why we are not geting the info form the cell db im not sure but i suspect it should not be supper invaisve to fix
17:44:23 rouk should i open a bug for tracking?
17:44:30 sean-k-mooney yes definetly
17:46:07 rouk alright, ill have that up once i have my kolla change deployed and verify my users are happy.
18:08:04 josephillips hey
18:08:21 rouk sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1881944
18:08:21 openstack Launchpad bug 1881944 in OpenStack Compute (nova) "nova-api returns empty block-device-mapping in metadata queries" [Undecided,New]
18:08:21 josephillips someone with idea of how to solve this The instance sync for host 'Computenode' did not match. Re-created its InstanceList.
18:08:27 josephillips happend eventualy
18:10:17 josephillips i had a issue with placement api that is already resolved
18:10:25 josephillips but this is the one error left on scheduler
18:20:41 artom mordred, hey, the functional tests in sdk - they hit a real devstack-deployed cloud in some of the zuul jobs in CI, right?
18:20:58 artom Or am I completely misunderstanding the code?
18:32:38 mordred artom: yup, that's right
18:33:08 artom mordred, cool, thanks. That was one of the points that came out of the discussion on the nova room this morning - how to test the SDK in CI
18:33:17 artom Good to know there's a framework already in place
18:33:31 artom Though I think no multinode testing, right? So live migration for instance isn't tested
18:33:33 mordred artom: all of the openstacksdk-functional-* jobs (and everything in openstack/tests/functional run against devstack
18:34:01 mordred artom: that's right - but I imagine if there is a multi-node devstack job that sets up multinode it would not be hard to add one
18:34:24 artom mordred, yeah, working on that now.
18:34:32 artom (If it seems disorganized, that's because it is)
18:34:42 mordred you're describing my life :)
18:34:53 artom (Expect a flurry of WIP patches as I organise my thoughts by working trough them)
18:35:05 mordred artom: looking forward to them
18:35:50 artom mordred, awesome, thanks
18:35:59 mordred artom: fwiw - we *always* run nova tests no matter what the devstack config is but we have a bunch of different devstack configs - so with neutron advanced services, with magnum, with senlin, with masakari etc
18:36:02 artom I gotta take the kiddos to the park first though :P
18:36:06 mordred but none of those configs don't have nova
18:36:39 artom mordred, that sounds reasonnable
18:39:42 dansmith artom: mordred: sounds like that's what we're looking for, and important to note that (apparently) sdk functional is more like nova integrated
18:51:33 mordred dansmith: english is hard
19:23:31 alex_xu sean-k-mooney: gibi sorry, just saw yulong's message, I tried chrome, but I think it is network issue. With intel proxy doesn't work, without proxy even can't open the page. but anyway we already switch to zoom :)
21:39:10 CeeMac Evening
21:42:49 CeeMac I'm trying to confirm some details on what exactly 'active' means in context to (I believe) how nova-usage reports cpu-hours, gb-hours etc to horizon for Admin | Overview | Usage Summary. My original understanding was that 'Active' equated to time instances spent in 'running' state it appears that may not be the case. Is anyone able to confirm as what documentation I could find wasn't clear.
21:43:39 CeeMac When looking at usage over a period e.g for the previous month.
21:57:42 CeeMac Digging in to the api it looks like 'active' may relate to 'server_usages.state' but I can't find a list of alternative values to give that context. For example is an instance 'active' when it is shutdown, does it need to be shelved or deleted to change from 'active' state and not accumulate usage values?
22:15:34 sean-k-mooney artom: if you need a multi node devstack just change the nodepool nodeset
22:16:02 sean-k-mooney the devstack ansiable playbooks support muiltnode so it shoudl be easy to make the jobs multinode
22:18:01 sean-k-mooney CeeMac: i think that is correct for CPUs but i think Ram and disk might be different
22:19:03 sean-k-mooney shelving is intented to be the way for tenant to consume less resouces when a vm is not activly needed
22:19:53 sean-k-mooney in a public cloud when an instnace is shelved you are normally only changed for the disk usage in glance and not the cpu/ram/devices consumed by the flavor
22:20:25 sean-k-mooney since shut down vms still consume the cpu/ram/disk from a billing/usage point of view its more or less the same as if its running
22:40:13 CeeMac Thanks sean-k-mooney, I've just found a list of states in the api-ref parameters file. Just need to work out where the logic is queried so I can validate which of the states are 'counted', but in the interim I'll look into shelving options and document that up for our users.
22:42:34 CeeMac I guess while the instance is 'there' its effectively holding a reservation against the CPU and ram resources? Hence its metered for prospective billing/accounting purposes.

Earlier   Later