Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-11
14:42:36 dansmith stephenfin: aight, well, when we moved tests/ into tests/unit things were not "smart", but whatevs
14:42:38 mriedem since we have explicit glance.py and cinder.py modules those are pretty clear what they are for
14:42:52 mriedem dansmith: oh yeah the tests -> tests/unit was another big one
14:43:19 efried mriedem: oh, I hope we're getting rid of the API shim
14:43:27 efried stephenfin: ? ^
14:43:34 mriedem efried: i'm fine with removing base_api.py and api.py (nova-net)
14:43:42 mriedem but i don't think we need to rename neutronv2.api to api.py
14:43:55 efried yeah, I'm fine leaving it named neutron
14:44:04 sean-k-mooney well on that how would people feel about eventurally moving the neutron code to os-vif?
14:44:34 mriedem why?
14:44:36 efried that sounds like a conversation for a far future release
14:44:57 sean-k-mooney well partly to not need to have any netowrking code in nova
14:44:59 mriedem "hey let's move some already really complicated and not very well understood code out to a library with a different core team"
14:45:10 canori01 Is it safe to rebuild the placement database? I have an issue where all my hypervisors are running into conflicts (conflicting resource provider name)
14:45:16 sean-k-mooney well all nova cores are os-vif cores
14:45:19 canori01 Can I just empty the db and bounce nova-compute?
14:45:27 sean-k-mooney but anyway its just an idea
14:45:33 mriedem sean-k-mooney: not worth the effort
14:45:38 mriedem pick your battles
14:45:56 sean-k-mooney its currently plugable so i was thinking of porting it then we could swap after
14:45:58 sean-k-mooney but ok
14:46:01 mriedem canori01: the inventory will rebuild itself automatically, the consumers/allocations will not
14:46:08 sean-k-mooney ill drop it for now
14:46:59 mriedem canori01: nova-manage placement heal_allocations should heal those up those if you do have to rebuild the placement db
14:47:13 mriedem *heal those up though
14:47:55 sean-k-mooney mriedem: that was added in rocky right. in queens we still had the periodic heal task form the move to placment
14:48:29 mriedem wrongish
14:48:57 canori01 mriedem: What happened is I used to have a third-party backup service that had service entries under nova. After removing them, all the hypervisors complain that an entry for them already exists and are unschedulable as a result
14:49:13 canori01 So my thought was to rebuild the placement db. I don't know if there's a better option
14:49:42 sean-k-mooney canori01: did you remvoe all the nova services
14:50:08 canori01 no, the nova services are still there
14:51:42 canori01 sean-k-mooney: would removing the service and bouncing nova-compute put things back in order?
14:53:07 mriedem kvm right?
14:53:23 mriedem the problem is placement has a unique constraint on the hostname, but the uuid on your computes has changed
14:53:41 mriedem the uuid on the compute_nodes table record that nova creates and uses to report the resource_providers to placement
14:55:19 mriedem so i think you're hitting some version of this https://bugs.launchpad.net/nova/+bug/1817833
14:55:19 openstack Launchpad bug 1817833 in OpenStack Compute (nova) "Check compute_id existence when nova-compute reports info to placement" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
14:56:20 sean-k-mooney canori01: no. i asked because if you remvoed the service the uuid would change when the agent restarts but the hostname would be the same and would cause a resouce provider conflcit
14:57:06 mriedem if you really need to rebuild the placement db, then i think your steps would be:
14:57:12 mriedem 1. backup your current placement db
14:57:20 mriedem 2. drop it and rebuild the schema so it's empty
14:57:34 mriedem 3. let the computes report their inventory in which will create resource providers on the first run
14:57:40 mriedem 4. run: nova-manage placement heal_allocations
14:57:42 stephenfin mriedem: replied on https://review.opendev.org/#/c/696509/
14:57:48 mriedem 5. run: nova-manage placement sync_aggregates
14:58:09 stephenfin tl;dr: if I don't do that req stuff, s*** breaks, so I did enough to make it work ¯\_(ツ)_/¯
14:58:13 mriedem this is also assuming you aren't using some of the more advanced features like QoS ports in neutron
14:58:43 canori01 mriedem: I am not yet using QoS for neutron
14:59:12 stephenfin mriedem: also, it's way up the stack but you should definitely look at https://review.opendev.org/#/c/696746/ since it affects your security group caching changes. I think what I did is correct
14:59:31 sean-k-mooney canori01: it specficlly woudl only be an issue if you were using minium bandwidth qos policy
14:59:32 mriedem jesus that is a big change
14:59:44 sean-k-mooney canori01: the other qos polices do not interact with placment
15:00:12 sean-k-mooney canori01: are you useing routed networks out of interest. e.g. calico
15:00:15 canori01 sean-k-mooney: So if I sync the uuid on the database to match the uuid of my existing service entries, that would also solve the issue?
15:00:28 canori01 For example, nova service-list has:
15:00:38 canori01 2c1037b3-4977-4a13-aea8-700a805cc11c | nova-compute | bctlz7nova36
15:01:05 sean-k-mooney canori01: that is easier said then done as you would have to also consider exitsting allocation too but in principal yes
15:01:06 canori01 placement has: | 2019-10-22 19:12:06 | 2019-11-25 19:27:36 | 157 | c53e4b12-0b0b-4eaa-9fb1-373da8538cea | bctlz7nova36
15:01:21 mriedem no those aren't the same
15:01:32 mriedem the nova services table uuid and nova compute_nodes uuid are not the same
15:01:41 canori01 ah ok
15:01:49 sean-k-mooney right the plcamment uuid is the compute node node uuid
15:02:13 canori01 sean-k-mooney: I'm not using calico. Just overlay vxlan networks advertised out with the neutron bgp agent
15:02:39 sean-k-mooney canori01: ok neutron report the network segment for routed networks to placmenet too
15:02:50 sean-k-mooney canori01: if you are using vxlan then you are fine
15:05:31 mriedem stephenfin: ok +2 on the 2nd from bottom change
15:05:40 stephenfin ta
15:05:58 efried stephenfin: and the bottom one is +A
15:06:06 canori01 mriedem: so would the safest course of action be to rebuild the placement db and heal the allocations?
15:06:09 mriedem i'm assuming canori01 didn't understand the question about what routed networks as a feature in neutron is
15:06:24 mriedem https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html
15:06:30 mriedem tl;dr it relies on aggregates
15:06:42 mriedem and they aren't supported in nova anyway so it's a red herring here
15:07:36 mriedem canori01: i think doing that (what i laid about above) is likely more fool proof than trying to hack the uuids to get all synced up
15:08:14 mriedem disclaimer: if you run into problems with that this isn't a support channel nor am i your paid vendor so i'm not going to be walking you through every issue you run into :)
15:08:16 efried It would be a good real world test of that procedure, anyway.
15:08:29 efried A canori in a coal mine, so to speak.
15:08:33 canori01 mriedem: ok, thanks. Also, I'm definitely not using the routed networks. My provider network is just one segment
15:08:36 mriedem efried: indeed - test it in (someone else's) production
15:09:28 mriedem efried: maybe a troubleshooting item to document, "oh no my placement db is all screwed up, how can i just start over w/ my existing nova"
15:09:30 canori01 mriedem: of course. I understand about the disclaimer :D
15:09:41 efried I thought cdent had that somewhere mebbe?
15:09:51 efried he's hanging out in -placement atm...
15:13:56 mriedem sean-k-mooney: to answer your earlier question, yes heal_allocations was added in rocky, but the RT did not report allocations peridiocially in rocky *unless* it's an ironic compute
15:14:13 mriedem see https://review.opendev.org/#/c/576462/
15:16:27 sean-k-mooney mriedem: no i ment it did that in queens
15:16:52 sean-k-mooney although only if you had ironic or pike compute nodes
15:17:06 mriedem correct
15:17:16 mriedem wait, no, <pike computes
15:17:29 sean-k-mooney am maybe
15:17:40 mriedem starting in pike once all your computes were upgraded we stopped having the RT report allocations because it would overwrite what the scheduler did and screw up allocations during move operations
15:17:58 sean-k-mooney yes
15:18:07 mriedem that's also why dansmith did migration-based allocatoins for move ops in queens
15:19:28 sean-k-mooney yep so basicaly the downstream issue was related to fixing allocation for undercloud (ironic) node where the customer acidentally deleted the service
15:20:34 mriedem stephenfin: while you're still around, i need you and efried to come to an agreement on https://review.opendev.org/#/c/696582/
15:20:34 sean-k-mooney because it was ironic and queens that perodic saved them. but they were asking what would happen if the same happend on the overcloud(libvirt nodes)
15:21:03 sean-k-mooney which was when we noticed that the heal_allcoation command was not on queens just rocky
15:34:59 mriedem speaking of which, melwitt - should i continue backporting these to rocky? https://review.opendev.org/#/q/topic:heal_allocations_dry_run+(status:open+OR+status:merged)
15:42:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Add troubleshooting doc about rebuilding the placement db https://review.opendev.org/698517
15:42:15 mriedem efried: canori01: ^ brain dump

Earlier   Later