Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-09
21:19:22 dansmith okay I see
21:19:46 dansmith I thought you were seeing consumer generation was null or zero or whatever in the third put, but still getting a conflict
21:19:49 dansmith but that makes sense now
21:20:03 mriedem Nov 06 19:48:37.013780 ubuntu-xenial-inap-mtl01-0000379614 nova-scheduler[12154]: WARNING nova.scheduler.client.report [None req-f266a0ff-2840-413d-9877-4500e61512f5 tempest-ServersNegativeTestJSON-477704048 tempest-ServersNegativeTestJSON-477704048] Failed to save allocation for 6665f00a-dcf1-4286-b075-d7dcd7c37487. Got HTTP 409: {"errors": [{"status": 409, "request_id": "req-c9ba6cbd-3b6e-4e5d-b550-9588be8a49d2", "code": "p
21:20:03 mriedem ment.concurrent_update", "detail": "There was a conflict when trying to complete your request.\n\n consumer generation conflict - expected null but got 1 ", "title": "Conflict"}]}
21:20:12 mriedem consumer generation conflict - expected null but got 1
21:20:20 mriedem yup - so new consumer but we're passing a generation of 1
21:20:25 mriedem from the old, now deleted consumer
21:21:07 dansmith cool
21:21:26 mriedem so,
21:21:42 dansmith I wish there was something better to communicate that, but any time we get "expected null" in that case, we should be able to re-try but as a non-move sort of thing
21:21:43 mriedem we can paper over this by deleting the allocations before marking the instance as shelved offloaded, but that's whack-a-moley
21:21:49 dansmith yeah
21:22:00 mriedem right we need to retry from claim_resources but i'm not sure what's the best way to do that
21:22:03 dansmith and like I said, I think it's not really any better, it just changes the problem
21:22:09 dansmith yeah
21:22:22 mriedem if we do retry that method, the next get for allocations will see there are none and we should be good
21:22:31 dansmith right
21:22:40 mriedem b/c we'll pass consumer_generation=None
21:22:59 mriedem i think i know what we can do
21:23:07 mriedem if we hit
21:23:07 mriedem if 'consumer generation conflict' in err['detail']:
21:23:13 mriedem we get the allocs again, and if empty,
21:23:15 mriedem we retry
21:23:17 mriedem easy peasy
21:23:22 mriedem it's a double get but meh?
21:23:29 dansmith yeah, it's just that it takes us an extra op,
21:23:34 dansmith when "expected null" should be enough
21:23:45 dansmith yeah
21:23:45 mriedem i can parse that out of the message if we want..
21:23:57 dansmith I know we can, it's just icky and unfortunate
21:23:59 mriedem with a TODO for more granular error codes later
21:24:02 dansmith like all the other cases in there
21:24:04 mriedem right
21:24:15 mriedem i haven't even started packing yet
21:24:24 mriedem laura is starting to check in on me every 30 minutes
21:24:31 mriedem "this is what i'm wearing all week! god!"
21:24:52 dansmith hah
21:25:35 mriedem plus my mother in law is here,
21:25:42 mriedem so lots of teenage angst memories coming back right now
21:25:51 mriedem the coffee and metallica doesn't hel[p
21:25:58 dansmith isn't that a good reason to pack and get out?
21:26:09 mriedem i've just holed up in my office
21:26:26 mriedem i'll crank out a patch for this bug and be off
21:45:39 openstackgerrit Jack Ding proposed openstack/nova master: Use virt.images.convert_image for qemu-img convert https://review.openstack.org/616692
21:52:35 fried_rice mriedem: Sanity check, please. The compute manager has a report client via the scheduler client, that's *not* the same as the report client the resource tracker has.
21:53:19 fried_rice which means my current SIGHUP doesn't do shit to the RT's cache
21:53:26 fried_rice I need to make the report client a singleton.
21:53:42 mriedem correct
21:53:46 mriedem we have report clients all over the place
21:53:50 mriedem api, conductor, scheduler
21:53:52 mriedem etc
21:54:14 fried_rice that's a scroo
21:54:55 fried_rice mriedem: So - make the report client a singleton (per process), or just diddle the compute manager's reset to hit the rt's reportclient instead.
21:55:59 fried_rice f, without knowing what the various ones in the compute manager are used for, is it really safe to make them a singleton?
21:56:04 mriedem the latter would be a smaller blast area
21:57:42 fried_rice I think I may have actually done this to myself, by removing that LazyLoader
21:57:50 fried_rice I suspect that guy was incidentally singleton-ing.
21:58:33 fried_rice nah, that should still have been creating separate instances per scheduler client.
22:03:56 dansmith fried_rice: yeah I thought that was making it a singleton a month ago when I was looking at it
22:04:18 dansmith a lot of stuff in nova used to be lazy loaded because.. um, terrible reasons
22:04:26 dansmith lazy loaded or pluggable
22:04:52 fried_rice dansmith: In this case it was supposedly because of a circular import. Whether that was ever really an issue, it isn't now, so I ripped it out. But having just looked, I still don't think it was making the report client a singleton. Care to confirm?
22:05:22 dansmith I think I confirmed that a month ago when I was looking into a seemingly recent memory leak
22:05:51 dansmith so I think it's fine that it's gone
22:06:07 dansmith I agree that randomly making it a singleton now should be done with care
22:06:23 dansmith but I don't really know how to convince myself that it's okay once its done, tbh
22:06:45 openstackgerrit Jack Ding proposed openstack/nova master: Use virt.images.convert_image for qemu-img convert https://review.openstack.org/616692
22:07:25 fried_rice well, using the RT's report client fixed the problem I was having. So maybe I pretend singleton was never suggested.
22:10:15 fried_rice oh, f, this is gonna break all over the place. I can't see a reason why the compute manager would possibly want or need to use separate report clients. I'd really like to put 'em together. If not making it a singleton, at least using only one of them from the compute manager.
22:10:46 fried_rice o/
22:15:46 mriedem fried_rice: the compute manager / RT using the same report client is probably fine,
22:16:06 mriedem a lot of that compute manager / RT code was cleaned up way back in ocata i think when jaypipes made the RT a singleton that tracked multiple compute nodes,
22:16:12 fried_rice mriedem: Ima put up an independent patch for that
22:16:12 mriedem whereas before it was 1 RT per compute node
22:16:14 fried_rice ah
22:16:33 mriedem they are very tightly coupled, like how the compute manager passes the virt driver into the RT
22:16:34 openstackgerrit Jack Ding proposed openstack/nova master: Use virt.images.convert_image for qemu-img convert https://review.openstack.org/616692
22:18:25 openstackgerrit Eric Fried proposed openstack/nova master: SIGHUP n-cpu to refresh provider tree cache https://review.openstack.org/615646
22:18:25 openstackgerrit Eric Fried proposed openstack/nova master: Reduce calls to placement from _ensure https://review.openstack.org/615677
22:18:26 openstackgerrit Eric Fried proposed openstack/nova master: Consolidate inventory refresh https://review.openstack.org/615695
22:18:26 openstackgerrit Eric Fried proposed openstack/nova master: Commonize _update code path https://review.openstack.org/615705
22:18:27 openstackgerrit Eric Fried proposed openstack/nova master: Turn off rp association refresh in nova-next https://review.openstack.org/616033
22:18:31 fried_rice let's see how that goes
22:21:44 fried_rice mriedem: Oh, my removal of lazyload probably reinstated "lockutils spam" mentioned in nova/compute/api.py@257
22:25:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Retry on consumer delete race in claim_resources https://review.openstack.org/617040
22:25:17 mriedem dansmith: jaypipes: fried_rice: ^ bingo bango
22:25:36 mriedem gibi: you too ^
22:25:48 mriedem the commit message is longer than the code
22:29:02 mriedem and with that i'm off
22:52:41 openstackgerrit Eric Fried proposed openstack/nova master: Rip the report client out of SchedulerClient https://review.openstack.org/617042
22:54:58 openstackgerrit Eric Fried proposed openstack/nova master: Rip the report client out of SchedulerClient https://review.openstack.org/617042
23:17:20 openstackgerrit Vlad Gusev proposed openstack/nova stable/pike: libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/604039
23:18:08 openstackgerrit Merged openstack/nova stable/pike: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599841
23:18:46 openstackgerrit Vlad Gusev proposed openstack/nova stable/pike: libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/604039
23:20:37 openstackgerrit Merged openstack/nova master: Add recreate test for bug 1799892 https://review.openstack.org/613304
23:20:37 openstack bug 1799892 in OpenStack Compute (nova) rocky "Placement API crashes with 500s in Rocky upgrade with downed compute nodes" [Medium,Triaged] https://launchpad.net/bugs/1799892
23:25:52 aspiers mriedem: thanks for the review! Regarding technical debt, my understanding is that the intention is very much for SUSE/AMD to carry on working to flesh out the functionality after implementation of the MVP described in the initial spec, rather than just to dump some half-baked implementation upstream and then vanish ;-) This would include adding support for attestation, migration etc.
23:26:04 aspiers ah, he's gone
23:27:55 openstackgerrit Vlad Gusev proposed openstack/nova stable/pike: libvirt: Use os.stat and os.path.getsize for RAW disk inspection https://review.openstack.org/607544

Earlier   Later