Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-11
15:01:32 mriedem the nova services table uuid and nova compute_nodes uuid are not the same
15:01:41 canori01 ah ok
15:01:49 sean-k-mooney right the plcamment uuid is the compute node node uuid
15:02:13 canori01 sean-k-mooney: I'm not using calico. Just overlay vxlan networks advertised out with the neutron bgp agent
15:02:39 sean-k-mooney canori01: ok neutron report the network segment for routed networks to placmenet too
15:02:50 sean-k-mooney canori01: if you are using vxlan then you are fine
15:05:31 mriedem stephenfin: ok +2 on the 2nd from bottom change
15:05:40 stephenfin ta
15:05:58 efried stephenfin: and the bottom one is +A
15:06:06 canori01 mriedem: so would the safest course of action be to rebuild the placement db and heal the allocations?
15:06:09 mriedem i'm assuming canori01 didn't understand the question about what routed networks as a feature in neutron is
15:06:24 mriedem https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html
15:06:30 mriedem tl;dr it relies on aggregates
15:06:42 mriedem and they aren't supported in nova anyway so it's a red herring here
15:07:36 mriedem canori01: i think doing that (what i laid about above) is likely more fool proof than trying to hack the uuids to get all synced up
15:08:14 mriedem disclaimer: if you run into problems with that this isn't a support channel nor am i your paid vendor so i'm not going to be walking you through every issue you run into :)
15:08:16 efried It would be a good real world test of that procedure, anyway.
15:08:29 efried A canori in a coal mine, so to speak.
15:08:33 canori01 mriedem: ok, thanks. Also, I'm definitely not using the routed networks. My provider network is just one segment
15:08:36 mriedem efried: indeed - test it in (someone else's) production
15:09:28 mriedem efried: maybe a troubleshooting item to document, "oh no my placement db is all screwed up, how can i just start over w/ my existing nova"
15:09:30 canori01 mriedem: of course. I understand about the disclaimer :D
15:09:41 efried I thought cdent had that somewhere mebbe?
15:09:51 efried he's hanging out in -placement atm...
15:13:56 mriedem sean-k-mooney: to answer your earlier question, yes heal_allocations was added in rocky, but the RT did not report allocations peridiocially in rocky *unless* it's an ironic compute
15:14:13 mriedem see https://review.opendev.org/#/c/576462/
15:16:27 sean-k-mooney mriedem: no i ment it did that in queens
15:16:52 sean-k-mooney although only if you had ironic or pike compute nodes
15:17:06 mriedem correct
15:17:16 mriedem wait, no, <pike computes
15:17:29 sean-k-mooney am maybe
15:17:40 mriedem starting in pike once all your computes were upgraded we stopped having the RT report allocations because it would overwrite what the scheduler did and screw up allocations during move operations
15:17:58 sean-k-mooney yes
15:18:07 mriedem that's also why dansmith did migration-based allocatoins for move ops in queens
15:19:28 sean-k-mooney yep so basicaly the downstream issue was related to fixing allocation for undercloud (ironic) node where the customer acidentally deleted the service
15:20:34 mriedem stephenfin: while you're still around, i need you and efried to come to an agreement on https://review.opendev.org/#/c/696582/
15:20:34 sean-k-mooney because it was ironic and queens that perodic saved them. but they were asking what would happen if the same happend on the overcloud(libvirt nodes)
15:21:03 sean-k-mooney which was when we noticed that the heal_allcoation command was not on queens just rocky
15:34:59 mriedem speaking of which, melwitt - should i continue backporting these to rocky? https://review.opendev.org/#/q/topic:heal_allocations_dry_run+(status:open+OR+status:merged)
15:42:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Add troubleshooting doc about rebuilding the placement db https://review.opendev.org/698517
15:42:15 mriedem efried: canori01: ^ brain dump
16:00:23 melwitt mriedem: if you do, it would be a help
16:00:34 melwitt I support ++
16:06:26 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support live migration with qos ports https://review.opendev.org/695905
16:06:45 gibi mriedem: the happy case support for live migration is now complete in ^^
16:10:19 efried stephenfin: re https://review.opendev.org/#/c/696582/ -- I want the shiny new command in the docs for sure. And I'm not sure your PS2 commentary meant you wanted it actually removed -- did it?
16:12:28 mriedem gibi: ack - throw that series into the runways etherpad?
16:12:43 mriedem i'm also waiting on efried to come back on https://review.opendev.org/#/c/696541/
16:13:25 efried mriedem: looking now
16:13:43 gibi mriedem: ack, adding...
16:14:36 mriedem gibi: i also replied to your comments on https://review.opendev.org/#/c/637070/ but then accidentally rebased
16:14:57 gibi mriedem: ack, put it in my queue
16:19:13 efried mriedem, gibi: I'm +2 on https://review.opendev.org/#/c/696541/
16:19:25 gibi efried: thanks
16:25:06 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove native LUKS compat code https://review.opendev.org/669121
16:25:15 mriedem i'm not
16:29:07 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Add --dry-run option to heal_allocations CLI https://review.opendev.org/698525
16:30:17 gibi mriedem: ack, I will need to get back to that patch tomorrow
16:31:28 gibi mriedem: most of the nois is there because this patch went through couple PSs with different solutions
16:31:38 gibi mriedem: I will get back to it tomorrow and clean it up
16:33:03 sean-k-mooney gibi: o/
16:38:25 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Add --instance option to heal_allocations https://review.opendev.org/698529
16:38:53 openstackgerrit sean mooney proposed openstack/nova stable/train: Block rebuild when NUMA topology changed https://review.opendev.org/698530
16:39:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.opendev.org/698531
16:40:54 openstackgerrit sean mooney proposed openstack/nova stable/train: Disable NUMATopologyFilter on rebuild https://review.opendev.org/698532
16:45:01 sean-k-mooney mriedem: since im backporting stuff should i backport https://review.opendev.org/#/c/695118/ which is the fix to https://bugs.launchpad.net/nova/+bug/1847367
16:45:01 openstack Launchpad bug 1847367 in OpenStack Compute (nova) "Images with hw:vif_multiqueue_enabled can be limited to 8 queues even if more are supported" [Undecided,Fix released] - Assigned to sean mooney (sean-k-mooney)
16:45:20 sean-k-mooney mriedem: it was opened against rocky so i guess it should go back at least that far
16:47:56 mriedem i'd probably let eandersson or his minions do the backports if they want them
16:49:01 efried mriedem: If I delete a shelved instance, does the virt driver ever get a crack at cleaning up?
16:49:01 efried a) if not offloaded, I assume yes, because the instance is still on the host
16:49:01 efried b) if offloaded, I assume no, because whose virt driver would we hit?
16:49:20 mriedem correct
16:49:25 efried thx
16:49:52 mriedem is this vpmem or accelerator related?
16:51:17 efried neither, vtpm
16:52:20 efried Means I think we're going to have to delete the swift obj from the conductor rather than the virt driver.
16:52:47 efried which kinda sucks because it's a virt driver-specific thing. At least the contents are.
16:52:51 mriedem i haven't been paying attentiong to the vtpm hullabaloo
16:52:58 mriedem *attention
16:53:17 mriedem conductor isn't involved in the server delete btw,
16:53:25 mriedem so the api would be doing whatever external cleanup is necessary
16:53:32 efried sigh, that's what I meant.
16:53:39 efried "controller"
16:53:47 efried can't imagine why I confuse that with "conductor".
16:54:02 efried though by now you would have thought I could get it the f right.
16:54:09 efried maybe after 8 years...
16:54:20 mriedem with enough pedantic ridicule you'll get there!
16:54:32 efried is that what it's for? gtk
16:54:51 efried I thought it was just plain old schoolyard bullying.
17:04:50 canori01 mriedem: That seems to have worked well. I'm still doing tests, but looks promising. I had to do it slightly differently. I'm running rocky, but my placement db is not broken out. So instead of dropping and recreating the db, I had to truncate the tables that placement uses. Then I bounced nova-compute and did the healing and aggregate syncing
17:07:35 mriedem ah cool
17:07:38 mriedem glad it's working
17:16:31 openstackgerrit Merged openstack/nova stable/queens: Do not update root_device_name during guest config https://review.opendev.org/696469
17:16:39 openstackgerrit Merged openstack/nova master: nova-net: Drop nova-network-base security group tests https://review.opendev.org/696508
17:19:15 sean-k-mooney efried: the virt driver should have cleaned up evertying on the host as part of the offload step
17:19:42 sean-k-mooney oh i see this is related to vtpm
17:24:20 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Resolve (most) flake8 3.x issues https://review.opendev.org/695732
17:24:21 openstackgerrit Stephen Finucane proposed openstack/nova master: Switch to flake8 3.x https://review.opendev.org/695733
17:31:32 mriedem wtf, so back on dec 5 i had a passing run of nova-multi-cell with migration tests enabled. now since the 9th with a new run all migration tests are failing because once the confirmed resized server is active again the api is saying the flavor is the old id even though i can see in the conductor logs where right before that the instance has the correct flavor

Earlier   Later