Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-11
14:56:20 sean-k-mooney canori01: no. i asked because if you remvoed the service the uuid would change when the agent restarts but the hostname would be the same and would cause a resouce provider conflcit
14:57:06 mriedem if you really need to rebuild the placement db, then i think your steps would be:
14:57:12 mriedem 1. backup your current placement db
14:57:20 mriedem 2. drop it and rebuild the schema so it's empty
14:57:34 mriedem 3. let the computes report their inventory in which will create resource providers on the first run
14:57:40 mriedem 4. run: nova-manage placement heal_allocations
14:57:42 stephenfin mriedem: replied on https://review.opendev.org/#/c/696509/
14:57:48 mriedem 5. run: nova-manage placement sync_aggregates
14:58:09 stephenfin tl;dr: if I don't do that req stuff, s*** breaks, so I did enough to make it work ¯\_(ツ)_/¯
14:58:13 mriedem this is also assuming you aren't using some of the more advanced features like QoS ports in neutron
14:58:43 canori01 mriedem: I am not yet using QoS for neutron
14:59:12 stephenfin mriedem: also, it's way up the stack but you should definitely look at https://review.opendev.org/#/c/696746/ since it affects your security group caching changes. I think what I did is correct
14:59:31 sean-k-mooney canori01: it specficlly woudl only be an issue if you were using minium bandwidth qos policy
14:59:32 mriedem jesus that is a big change
14:59:44 sean-k-mooney canori01: the other qos polices do not interact with placment
15:00:12 sean-k-mooney canori01: are you useing routed networks out of interest. e.g. calico
15:00:15 canori01 sean-k-mooney: So if I sync the uuid on the database to match the uuid of my existing service entries, that would also solve the issue?
15:00:28 canori01 For example, nova service-list has:
15:00:38 canori01 2c1037b3-4977-4a13-aea8-700a805cc11c | nova-compute | bctlz7nova36
15:01:05 sean-k-mooney canori01: that is easier said then done as you would have to also consider exitsting allocation too but in principal yes
15:01:06 canori01 placement has: | 2019-10-22 19:12:06 | 2019-11-25 19:27:36 | 157 | c53e4b12-0b0b-4eaa-9fb1-373da8538cea | bctlz7nova36
15:01:21 mriedem no those aren't the same
15:01:32 mriedem the nova services table uuid and nova compute_nodes uuid are not the same
15:01:41 canori01 ah ok
15:01:49 sean-k-mooney right the plcamment uuid is the compute node node uuid
15:02:13 canori01 sean-k-mooney: I'm not using calico. Just overlay vxlan networks advertised out with the neutron bgp agent
15:02:39 sean-k-mooney canori01: ok neutron report the network segment for routed networks to placmenet too
15:02:50 sean-k-mooney canori01: if you are using vxlan then you are fine
15:05:31 mriedem stephenfin: ok +2 on the 2nd from bottom change
15:05:40 stephenfin ta
15:05:58 efried stephenfin: and the bottom one is +A
15:06:06 canori01 mriedem: so would the safest course of action be to rebuild the placement db and heal the allocations?
15:06:09 mriedem i'm assuming canori01 didn't understand the question about what routed networks as a feature in neutron is
15:06:24 mriedem https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html
15:06:30 mriedem tl;dr it relies on aggregates
15:06:42 mriedem and they aren't supported in nova anyway so it's a red herring here
15:07:36 mriedem canori01: i think doing that (what i laid about above) is likely more fool proof than trying to hack the uuids to get all synced up
15:08:14 mriedem disclaimer: if you run into problems with that this isn't a support channel nor am i your paid vendor so i'm not going to be walking you through every issue you run into :)
15:08:16 efried It would be a good real world test of that procedure, anyway.
15:08:29 efried A canori in a coal mine, so to speak.
15:08:33 canori01 mriedem: ok, thanks. Also, I'm definitely not using the routed networks. My provider network is just one segment
15:08:36 mriedem efried: indeed - test it in (someone else's) production
15:09:28 mriedem efried: maybe a troubleshooting item to document, "oh no my placement db is all screwed up, how can i just start over w/ my existing nova"
15:09:30 canori01 mriedem: of course. I understand about the disclaimer :D
15:09:41 efried I thought cdent had that somewhere mebbe?
15:09:51 efried he's hanging out in -placement atm...
15:13:56 mriedem sean-k-mooney: to answer your earlier question, yes heal_allocations was added in rocky, but the RT did not report allocations peridiocially in rocky *unless* it's an ironic compute
15:14:13 mriedem see https://review.opendev.org/#/c/576462/
15:16:27 sean-k-mooney mriedem: no i ment it did that in queens
15:16:52 sean-k-mooney although only if you had ironic or pike compute nodes
15:17:06 mriedem correct
15:17:16 mriedem wait, no, <pike computes
15:17:29 sean-k-mooney am maybe
15:17:40 mriedem starting in pike once all your computes were upgraded we stopped having the RT report allocations because it would overwrite what the scheduler did and screw up allocations during move operations
15:17:58 sean-k-mooney yes
15:18:07 mriedem that's also why dansmith did migration-based allocatoins for move ops in queens
15:19:28 sean-k-mooney yep so basicaly the downstream issue was related to fixing allocation for undercloud (ironic) node where the customer acidentally deleted the service
15:20:34 sean-k-mooney because it was ironic and queens that perodic saved them. but they were asking what would happen if the same happend on the overcloud(libvirt nodes)
15:20:34 mriedem stephenfin: while you're still around, i need you and efried to come to an agreement on https://review.opendev.org/#/c/696582/
15:21:03 sean-k-mooney which was when we noticed that the heal_allcoation command was not on queens just rocky
15:34:59 mriedem speaking of which, melwitt - should i continue backporting these to rocky? https://review.opendev.org/#/q/topic:heal_allocations_dry_run+(status:open+OR+status:merged)
15:42:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Add troubleshooting doc about rebuilding the placement db https://review.opendev.org/698517
15:42:15 mriedem efried: canori01: ^ brain dump
16:00:23 melwitt mriedem: if you do, it would be a help
16:00:34 melwitt I support ++
16:06:26 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support live migration with qos ports https://review.opendev.org/695905
16:06:45 gibi mriedem: the happy case support for live migration is now complete in ^^
16:10:19 efried stephenfin: re https://review.opendev.org/#/c/696582/ -- I want the shiny new command in the docs for sure. And I'm not sure your PS2 commentary meant you wanted it actually removed -- did it?
16:12:28 mriedem gibi: ack - throw that series into the runways etherpad?
16:12:43 mriedem i'm also waiting on efried to come back on https://review.opendev.org/#/c/696541/
16:13:25 efried mriedem: looking now
16:13:43 gibi mriedem: ack, adding...
16:14:36 mriedem gibi: i also replied to your comments on https://review.opendev.org/#/c/637070/ but then accidentally rebased
16:14:57 gibi mriedem: ack, put it in my queue
16:19:13 efried mriedem, gibi: I'm +2 on https://review.opendev.org/#/c/696541/
16:19:25 gibi efried: thanks
16:25:06 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove native LUKS compat code https://review.opendev.org/669121
16:25:15 mriedem i'm not
16:29:07 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Add --dry-run option to heal_allocations CLI https://review.opendev.org/698525
16:30:17 gibi mriedem: ack, I will need to get back to that patch tomorrow
16:31:28 gibi mriedem: most of the nois is there because this patch went through couple PSs with different solutions
16:31:38 gibi mriedem: I will get back to it tomorrow and clean it up
16:33:03 sean-k-mooney gibi: o/
16:38:25 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Add --instance option to heal_allocations https://review.opendev.org/698529
16:38:53 openstackgerrit sean mooney proposed openstack/nova stable/train: Block rebuild when NUMA topology changed https://review.opendev.org/698530
16:39:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.opendev.org/698531
16:40:54 openstackgerrit sean mooney proposed openstack/nova stable/train: Disable NUMATopologyFilter on rebuild https://review.opendev.org/698532
16:45:01 openstack Launchpad bug 1847367 in OpenStack Compute (nova) "Images with hw:vif_multiqueue_enabled can be limited to 8 queues even if more are supported" [Undecided,Fix released] - Assigned to sean mooney (sean-k-mooney)
16:45:01 sean-k-mooney mriedem: since im backporting stuff should i backport https://review.opendev.org/#/c/695118/ which is the fix to https://bugs.launchpad.net/nova/+bug/1847367
16:45:20 sean-k-mooney mriedem: it was opened against rocky so i guess it should go back at least that far
16:47:56 mriedem i'd probably let eandersson or his minions do the backports if they want them
16:49:01 efried b) if offloaded, I assume no, because whose virt driver would we hit?
16:49:01 efried a) if not offloaded, I assume yes, because the instance is still on the host
16:49:01 efried mriedem: If I delete a shelved instance, does the virt driver ever get a crack at cleaning up?
16:49:20 mriedem correct
16:49:25 efried thx
16:49:52 mriedem is this vpmem or accelerator related?
16:51:17 efried neither, vtpm
16:52:20 efried Means I think we're going to have to delete the swift obj from the conductor rather than the virt driver.
16:52:47 efried which kinda sucks because it's a virt driver-specific thing. At least the contents are.

Earlier   Later