Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-11
22:58:03 mriedem sure, but people are going to use knobs if we give them
22:58:25 mriedem i'm not saying that's what osa / chef are hitting here, but it's another possibility
22:58:29 eandersson Unfortunately a lot of changes between master and rocky in that script
22:58:46 mriedem eandersson: oh - probably the report client refactoring stuff...
22:58:53 dansmith melwitt: it's not so bad, it just seems like a bad idea to act like that's okay or expected.. we're looking for un-mapped records and adding host mappings, then marking those service records as mapped
22:59:05 mriedem eandersson: if we had a bug or something we could think about backporting that change upstream
22:59:38 mriedem heh ibm loves the email
22:59:38 dansmith melwitt: if you make sure none of that gets skipped (like one gets set mapped without a mapping being created, etc) then it's okay I guess
23:00:01 efried pretty sure they're bouncing 'em, cause I got a snail mail letter from another thing where I hadn't changed it.
23:00:24 dansmith melwitt: and the scheduler periodic is a case where we're kinda inviting you to run multiple in parallel, so there's that
23:00:34 melwitt dansmith: yeah... I could see that too. at the same time, I could see two discover_hosts happening to overlap. and yeah would have to be done with care
23:01:08 dansmith melwitt: but if we just skip that one and keep scanning, then you could have multiples of those just hammering your database when you bring a bunch of nodes online, all but one of them losing on every attempt
23:01:22 melwitt I was just thinking about the lock thing and wondered about ignoring dupes
23:01:31 dansmith if they all backoff and stop, then the one that won will proceed
23:01:37 dansmith the lock thing is just a hack to handle the sloppy ansible case
23:01:42 dansmith or puppet or whatever
23:01:44 melwitt yeah
23:02:08 dansmith since the have the same case for other things in nova-manage,
23:02:31 dansmith like db_sync, online_data_migrations, etc, it seems like we should just prescribe that those are to be run singly
23:02:44 dansmith and if not, we just need to fix it all, otherwise we're inviting confusion
23:02:46 melwitt that's true
23:03:08 dansmith heck, archive and purge probably explode if you run them in parallel
23:03:18 dansmith and probably map_instances
23:03:26 melwitt haha yeah
23:03:46 melwitt so the only valid concern would be the periodics
23:04:16 dansmith yeah, the periodic is legit, although like I said we added that for tripleo undercloud where they didn't have instrumentation to even run it,
23:04:21 dansmith and they only have one controller node
23:04:33 melwitt and I guess that would resolve itself eventually as the periodics keep running?
23:04:47 dansmith so we could *also* just augment the help for that and place a warning and/or make sure it just logs a warning and doesn't make too much noise in the logs
23:04:53 dansmith yes
23:05:03 melwitt like you fail by bad luck and then next time you'll get it
23:05:06 dansmith right
23:05:29 dansmith it's a slow lazy discovery anyway, so who cares, and if one fails, the other likely succeeded and mapped everything anyway
23:05:31 melwitt yeah, I definitely wanted to make docs/usage update to help with this somehow
23:05:58 dansmith jesus, is there no mystique in the art of running a nova these days?
23:05:59 melwitt yeah
23:06:03 dansmith always with the documentation
23:06:27 melwitt lol, mystique
23:17:26 openstackgerrit sean mooney proposed openstack/nova master: extend libvirt video model support https://review.openstack.org/647733
23:23:35 openstackgerrit Merged openstack/nova stable/stein: Update instance.availability_zone on revertResize https://review.openstack.org/648402
23:41:24 zzzeek efried: hey
23:47:07 openstackgerrit Merged openstack/nova stable/stein: Temporarily mutate migration object in finish_revert_resize https://review.openstack.org/648688
#openstack-nova - 2019-04-12
00:16:49 openstackgerrit Tony Breeds proposed openstack/nova master: Add pep8 to test-requirements https://review.openstack.org/651944
00:16:49 openstackgerrit Tony Breeds proposed openstack/nova master: Uncap jsonschema https://review.openstack.org/651943
00:17:24 tonyb efried, stephenfin: ^^ the first is unrelated to your hacking work but is required
00:17:54 tonyb efried, stephenfin: the second just makes it explict and clear why we're adding pep8
00:20:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Add --instance option to heal_allocations https://review.openstack.org/651945
00:20:29 mriedem ceryx: eandersson: ^
00:20:37 mriedem if you want to just work on that one problem instance
00:20:45 mriedem zzzeek: we figured it out
00:40:46 openstackgerrit Merged openstack/python-novaclient stable/stein: Revert "Fix crashing console-log" https://review.openstack.org/650363
00:41:30 mriedem efried: i just realized that this is always True https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L403
00:41:37 mriedem because Flavor.extra_specs is always loaded
00:41:43 mriedem it doesn't mean extra_specs is not empty
00:42:13 mriedem and that's extremely important because for a volume-backed server, we could otherwise be trying to PUT allocations with DISK_GB=0
00:42:15 mriedem which is a no-no
00:43:51 openstackgerrit melanie witt proposed openstack/nova master: Warn for duplicate host mappings during discover_hosts https://review.openstack.org/651947
00:45:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.openstack.org/651948
00:47:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.openstack.org/651948
00:49:24 eandersson We removed the dups in the allocation table and re-ran the script and it worked
00:51:06 eandersson but that would have been neat
00:51:54 mriedem eandersson: ok so now you just have the 2 allocations records for the one instance consumer against the same compute node resource provider that you'd expect for that instance?
00:52:12 eandersson Yea
00:52:15 mriedem i.e. instance.node matches the compute_nodes table record matches the rp
00:52:17 mriedem ok cool
00:52:37 mriedem so i'm still guessing the multiple allocations against multiple providers was a result of failed migrations where the allocations - created by the scheduler - weren't cleaned up properly
00:52:50 eandersson and the actual allocation was never added I think
00:53:02 eandersson So it failed to clean up, but also failed to add the last allocation
00:53:03 mriedem eandersson: i.e. https://review.openstack.org/#/c/647566/
00:53:15 mriedem i'm not sure what you mean by 'last allocation'
00:53:34 eandersson as the compute it ended up on was not actually in the allocation table
00:54:02 eandersson but my colleague can provide more detailed information tomorrow
00:54:04 mriedem unless i'm missing something i don't think that can happen
00:54:17 mriedem the scheduler will create the allocations - if that fails, you'd get novalidhost most likely
00:54:32 mriedem that or we'd be rescheduling to another host
00:55:03 mriedem but we could have definitely been leaking allocations somehow when we fail ala https://bugs.launchpad.net/nova/+bug/1821594
00:55:06 openstack Launchpad bug 1821594 in OpenStack Compute (nova) stein "Error in confirm_migration leaves stale allocations and 'confirming' migration state" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
00:55:17 mriedem would be good to nail down if you could figure out what actually failed though to see if we have another leak
00:55:43 mriedem i still need to get the fix for that backported to rocky
00:57:28 mriedem gotta go - time to blow snow
00:57:54 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311
00:58:08 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581
00:58:25 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017
00:58:47 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018
00:59:06 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019
00:59:34 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.openstack.org/576020
01:00:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.openstack.org/576027
01:01:03 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.openstack.org/576031
01:01:27 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299
01:01:54 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344
01:02:17 eandersson mriedem_afk, you are right - I assumed the compute service id in cli would match resource_provider_id
01:02:36 eandersson So the above bug would explain it
01:02:44 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.openstack.org/576673
01:03:13 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.openstack.org/576676
01:03:37 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.openstack.org/576689
01:04:09 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.openstack.org/576709
01:04:40 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.openstack.org/576712
01:05:16 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.openstack.org/641200
01:05:32 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add minimum value in max_concurrent_live_migrations https://review.openstack.org/648302
01:06:45 openstackgerrit melanie witt proposed openstack/nova master: Warn for duplicate host mappings during discover_hosts https://review.openstack.org/651947

Earlier   Later