Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-11
22:39:12 efried unique constraint or no
22:39:27 efried what, are we hitting that NotFound from multiple threads, reliably, at the same time??
22:44:16 mriedem i'm able to recreate the same thing in the db in my devstack http://paste.openstack.org/show/749218/
22:44:22 mriedem but i'm not hitting errors running the online data migration
22:44:34 mriedem i created a server and ran the migrations a few times
22:45:50 mriedem imacdonn_: are you running the CLI concurrently or something? or able to recreate manually?
22:46:39 imacdonn_ mriedem, you mean like two instances of nova-manage at the same time? no.... I can run it manually and it fails consistently
22:46:58 mriedem on a fresh install?
22:47:54 imacdonn_ One of these was a fresh install, but I did create an instance at some point
22:51:36 mriedem ok i'm not sure how to recreate it then
22:52:12 mriedem ceryx: eandersson: it's getting late here for the work day but please follow up with me on whatever you figure out with this allocations thing
22:52:45 melwitt dansmith: would another option other than trying to lock for discover_hosts be to try-except around the host_mapping.create() and catch and ignore DBDuplicateEntry?
22:54:01 melwitt https://github.com/openstack/nova/blob/master/nova/objects/host_mapping.py#L193
22:54:33 mriedem melwitt: here as well https://github.com/openstack/nova/blob/master/nova/objects/host_mapping.py#L211
22:54:42 mriedem that seems reasonable though
22:54:49 melwitt yes, two places
22:55:37 ceryx mriedem: Thanks, will do. eandersson is building a new container now with that patch to test.
22:55:57 dansmith melwitt: obviously that will avoid the trace, yeah. I'd still abort and say "looks like you're doing naughty things, so I'm stopping"
22:56:20 mriedem ceryx: unfortunately there isn't a way to just run heal_allocations on a specific instance yet, but maybe you won't get too much output
22:56:27 melwitt dansmith: hm, ok
22:56:48 mriedem the discover hosts periodic is in the scheduler process right?
22:57:07 mriedem so you could have multiple schedulers running discover_hosts at the same time
22:57:07 dansmith mriedem: yup
22:57:19 dansmith yup
22:57:29 mriedem so...probably not a terrible idea to just handle the duplicate and move on
22:57:34 melwitt I guess I'm not sure why it's so bad because the database will synchronize everything anyway. if there's a collision, just ignore it
22:57:42 dansmith that periodic was really just for the ironic case where you'd have a small control plane
22:58:03 mriedem sure, but people are going to use knobs if we give them
22:58:25 mriedem i'm not saying that's what osa / chef are hitting here, but it's another possibility
22:58:29 eandersson Unfortunately a lot of changes between master and rocky in that script
22:58:46 mriedem eandersson: oh - probably the report client refactoring stuff...
22:58:53 dansmith melwitt: it's not so bad, it just seems like a bad idea to act like that's okay or expected.. we're looking for un-mapped records and adding host mappings, then marking those service records as mapped
22:59:05 mriedem eandersson: if we had a bug or something we could think about backporting that change upstream
22:59:38 dansmith melwitt: if you make sure none of that gets skipped (like one gets set mapped without a mapping being created, etc) then it's okay I guess
22:59:38 mriedem heh ibm loves the email
23:00:01 efried pretty sure they're bouncing 'em, cause I got a snail mail letter from another thing where I hadn't changed it.
23:00:24 dansmith melwitt: and the scheduler periodic is a case where we're kinda inviting you to run multiple in parallel, so there's that
23:00:34 melwitt dansmith: yeah... I could see that too. at the same time, I could see two discover_hosts happening to overlap. and yeah would have to be done with care
23:01:08 dansmith melwitt: but if we just skip that one and keep scanning, then you could have multiples of those just hammering your database when you bring a bunch of nodes online, all but one of them losing on every attempt
23:01:22 melwitt I was just thinking about the lock thing and wondered about ignoring dupes
23:01:31 dansmith if they all backoff and stop, then the one that won will proceed
23:01:37 dansmith the lock thing is just a hack to handle the sloppy ansible case
23:01:42 dansmith or puppet or whatever
23:01:44 melwitt yeah
23:02:08 dansmith since the have the same case for other things in nova-manage,
23:02:31 dansmith like db_sync, online_data_migrations, etc, it seems like we should just prescribe that those are to be run singly
23:02:44 dansmith and if not, we just need to fix it all, otherwise we're inviting confusion
23:02:46 melwitt that's true
23:03:08 dansmith heck, archive and purge probably explode if you run them in parallel
23:03:18 dansmith and probably map_instances
23:03:26 melwitt haha yeah
23:03:46 melwitt so the only valid concern would be the periodics
23:04:16 dansmith yeah, the periodic is legit, although like I said we added that for tripleo undercloud where they didn't have instrumentation to even run it,
23:04:21 dansmith and they only have one controller node
23:04:33 melwitt and I guess that would resolve itself eventually as the periodics keep running?
23:04:47 dansmith so we could *also* just augment the help for that and place a warning and/or make sure it just logs a warning and doesn't make too much noise in the logs
23:04:53 dansmith yes
23:05:03 melwitt like you fail by bad luck and then next time you'll get it
23:05:06 dansmith right
23:05:29 dansmith it's a slow lazy discovery anyway, so who cares, and if one fails, the other likely succeeded and mapped everything anyway
23:05:31 melwitt yeah, I definitely wanted to make docs/usage update to help with this somehow
23:05:58 dansmith jesus, is there no mystique in the art of running a nova these days?
23:05:59 melwitt yeah
23:06:03 dansmith always with the documentation
23:06:27 melwitt lol, mystique
23:17:26 openstackgerrit sean mooney proposed openstack/nova master: extend libvirt video model support https://review.openstack.org/647733
23:23:35 openstackgerrit Merged openstack/nova stable/stein: Update instance.availability_zone on revertResize https://review.openstack.org/648402
23:41:24 zzzeek efried: hey
23:47:07 openstackgerrit Merged openstack/nova stable/stein: Temporarily mutate migration object in finish_revert_resize https://review.openstack.org/648688
#openstack-nova - 2019-04-12
00:16:49 openstackgerrit Tony Breeds proposed openstack/nova master: Uncap jsonschema https://review.openstack.org/651943
00:16:49 openstackgerrit Tony Breeds proposed openstack/nova master: Add pep8 to test-requirements https://review.openstack.org/651944
00:17:24 tonyb efried, stephenfin: ^^ the first is unrelated to your hacking work but is required
00:17:54 tonyb efried, stephenfin: the second just makes it explict and clear why we're adding pep8
00:20:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Add --instance option to heal_allocations https://review.openstack.org/651945
00:20:29 mriedem ceryx: eandersson: ^
00:20:37 mriedem if you want to just work on that one problem instance
00:20:45 mriedem zzzeek: we figured it out
00:40:46 openstackgerrit Merged openstack/python-novaclient stable/stein: Revert "Fix crashing console-log" https://review.openstack.org/650363
00:41:30 mriedem efried: i just realized that this is always True https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L403
00:41:37 mriedem because Flavor.extra_specs is always loaded
00:41:43 mriedem it doesn't mean extra_specs is not empty
00:42:13 mriedem and that's extremely important because for a volume-backed server, we could otherwise be trying to PUT allocations with DISK_GB=0
00:42:15 mriedem which is a no-no
00:43:51 openstackgerrit melanie witt proposed openstack/nova master: Warn for duplicate host mappings during discover_hosts https://review.openstack.org/651947
00:45:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.openstack.org/651948
00:47:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.openstack.org/651948
00:49:24 eandersson We removed the dups in the allocation table and re-ran the script and it worked
00:51:06 eandersson but that would have been neat
00:51:54 mriedem eandersson: ok so now you just have the 2 allocations records for the one instance consumer against the same compute node resource provider that you'd expect for that instance?
00:52:12 eandersson Yea
00:52:15 mriedem i.e. instance.node matches the compute_nodes table record matches the rp
00:52:17 mriedem ok cool
00:52:37 mriedem so i'm still guessing the multiple allocations against multiple providers was a result of failed migrations where the allocations - created by the scheduler - weren't cleaned up properly
00:52:50 eandersson and the actual allocation was never added I think
00:53:02 eandersson So it failed to clean up, but also failed to add the last allocation
00:53:03 mriedem eandersson: i.e. https://review.openstack.org/#/c/647566/
00:53:15 mriedem i'm not sure what you mean by 'last allocation'
00:53:34 eandersson as the compute it ended up on was not actually in the allocation table
00:54:02 eandersson but my colleague can provide more detailed information tomorrow
00:54:04 mriedem unless i'm missing something i don't think that can happen

Earlier   Later