| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-11-01 | |||
| 14:17:47 | mriedem | it fails because we don't find the instance allocation for some reason http://logs.openstack.org/08/516708/3/gate/legacy-tempest-dsvm-py35/4d8d6a3/logs/screen-n-super-cond.txt.gz#_Oct_31_23_18_04_391235 | |
| 14:20:35 | mriedem | i can see the allocations created for that instance here http://logs.openstack.org/08/516708/3/gate/legacy-tempest-dsvm-py35/4d8d6a3/logs/screen-placement-api.txt.gz#_Oct_31_23_18_00_637846 | |
| 14:20:59 | mriedem | but shortly after that something deletes the instance allocations http://logs.openstack.org/08/516708/3/gate/legacy-tempest-dsvm-py35/4d8d6a3/logs/screen-placement-api.txt.gz#_Oct_31_23_18_00_805083 | |
| 14:25:13 | efried | mriedem Opinion on something? | |
| 14:25:38 | mriedem | ok | |
| 14:25:53 | efried | In the work for granular resource requests, we're finding ourselves needing a very similar POD obj on either side of the placement API barrier. | |
| 14:26:23 | mriedem | what is a POD object? | |
| 14:26:31 | efried | plain ol' data | |
| 14:26:49 | efried | Question is: do we duplicate the base class definition on both sides, define it in one place? | |
| 14:27:15 | efried | If the latter, where would that be, given the long-term goal of separating out the placement API. | |
| 14:27:51 | mriedem | is it a versioned object? | |
| 14:28:24 | efried | No. It doesn't go across the wire. It's just used to represent the same concept on either side. | |
| 14:28:26 | efried | In real terms: https://review.openstack.org/#/c/515151/10/nova/scheduler/utils.py@47 and https://review.openstack.org/#/c/514091/6/nova/api/openstack/placement/util.py@36 | |
| 14:28:58 | efried | On the flavor side, it gets built by parsing extra_specs. On the placement side, it gets built by parsing the query string. | |
| 14:29:59 | mriedem | what you have is probably fine isn't it? they aren't exactly identical | |
| 14:30:23 | mriedem | i don't think we're going to create a library for a single object to share between placement and nova right now, maybe when the split happens and there is more common code to share | |
| 14:31:15 | mriedem | there is also nova/common/ | |
| 14:31:19 | efried | mriedem Yeah, they're not exactly identical, though they easily could be made so. | |
| 14:31:41 | mriedem | idk, throw them into nova/common/ if you want a single object right now | |
| 14:32:07 | mriedem | nova/common/placement.py? | |
| 14:32:22 | efried | I'm fine keeping them separated - philosophically I would think it's not at all uncommon for identical or almost-identical classes to exist on either side of an API boundary. | |
| 14:32:52 | efried | I mean, if this were e.g. the neutron API we were talking to, we would at best be talking about importing something from a neutron lib. | |
| 14:33:55 | efried | Anyway, jaypipes was on the fence. Sounds like you also don't have a super strong bias either way. | |
| 14:35:46 | mriedem | yeah i don't | |
| 14:36:10 | openstackgerrit | Eric Fried proposed openstack/nova master: placement: Parse granular resources & traits https://review.openstack.org/514091 | |
| 14:36:25 | efried | mriedem Thanks. | |
| 14:38:11 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Numbered groupings to GET /allocation_candidates https://review.openstack.org/514092 | |
| 14:42:02 | efried | jaypipes Just noticed that the filtery part of nova.objects.resource_provider.ResourceProviderList#_get_all_by_filters_from_db is (or should be) common with your _get_provider_ids_matching_all | |
| 14:42:23 | efried | jaypipes (still haven't finished the series, so maybe you already factored that out.) | |
| 14:46:47 | efried | alex_xu Still around? | |
| 14:49:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Raise specific exception when swapping migration allocations fails https://review.openstack.org/517004 | |
| 14:49:25 | dansmith | mriedem: hmm, you think there's a bug in that delete-if-deleted logic | |
| 14:49:38 | mriedem | dansmith: yes, that's what's deleting the allocation | |
| 14:49:42 | mriedem | http://logs.openstack.org/08/516708/3/gate/legacy-tempest-dsvm-py35/4d8d6a3/logs/screen-n-cpu.txt.gz#_Oct_31_23_18_00_780729 | |
| 14:49:57 | mriedem | https://review.openstack.org/517004 fixes the misleading 400 out of the API | |
| 14:51:29 | mriedem | dansmith: so this must be a race between the time the RT tracks the instance as a 'known instance' and the time the update_available_resource periodic runs | |
| 14:52:15 | dansmith | mriedem: it's not supposed to delete it unless it's really deleted=yes though | |
| 14:52:16 | mriedem | yeah the periodic starts here http://logs.openstack.org/08/516708/3/gate/legacy-tempest-dsvm-py35/4d8d6a3/logs/screen-n-cpu.txt.gz#_Oct_31_23_18_00_165850 | |
| 14:52:41 | dansmith | mriedem: although, I wonder if it might create the allocation before it's created in the cell db and thus it's a BR only, compute thinks it's deleted like archived | |
| 14:52:56 | mriedem | eesh | |
| 14:54:02 | mriedem | yeah i guess (1) scheduler creates allocation, (2) periodic on compute starts - gets allocations against itself, deletes allocations b/c InstanceNotFound in cell (3) superconductor creates instance in cell for the selected host | |
| 14:54:14 | mriedem | shite | |
| 14:54:43 | dansmith | yeah | |
| 14:54:44 | mriedem | and...we can't really have the compute try to find out if a build request exists can we given that's API DB and the compute shouldn't have access to the API DB | |
| 14:54:55 | dansmith | nope | |
| 14:55:01 | mriedem | i mean, i guess if you're ocata or pike single-cell that might be something you're still doing | |
| 14:55:04 | mriedem | for the late affinity check | |
| 14:55:26 | mriedem | well i'll open a bug to track it anyway | |
| 14:55:28 | dansmith | we've never had computes use objects that are only on the api db, AFAIK, | |
| 14:55:41 | mriedem | server group is in the api db | |
| 14:55:45 | mriedem | *InstanceGroup | |
| 14:55:53 | dansmith | well, because it moved | |
| 14:56:08 | mriedem | yeah | |
| 14:56:08 | dansmith | anyway, that's a door we don't want to open, even under config I think | |
| 14:56:32 | dansmith | deleting only if we find an actually-deleted instance would be one way to handle it, | |
| 14:56:43 | dansmith | which pushes the race to the delete-archive boundary | |
| 14:56:48 | dansmith | but that's much better, IMHO | |
| 14:57:17 | dansmith | if they're doing fast purging (like immediate purging of all deleted things, then they could bump into it, but much less likely, and we could have nova-manage check for allocations potentially | |
| 15:01:54 | openstackgerrit | Matthew Booth proposed openstack/nova master: libvirt: Don't VIR_MIGRATE_NON_SHARED_INC without migrate_disks https://review.openstack.org/507202 | |
| 15:02:20 | openstackgerrit | Matthew Booth proposed openstack/nova master: DNM: Run test_volume_backed_live_migration and iscsi test https://review.openstack.org/508163 | |
| 15:03:31 | openstackgerrit | Jianghua Wang proposed openstack/nova master: XenAPI: get vGPU stats from hypervisor https://review.openstack.org/512965 | |
| 15:05:16 | jianghuaw | jaypipes, thanks for the comment. Yes, you're correct. The above is the reworked revision. | |
| 15:06:20 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/511035 | |
| 15:06:22 | mriedem | dansmith: ok here is the bug https://bugs.launchpad.net/nova/+bug/1729371 | |
| 15:06:23 | openstack | Launchpad bug 1729371 in OpenStack Compute (nova) "ResourceTracker races to delete instance allocations before instance is mapped to a cell" [High,Triaged] | |
| 15:06:41 | dansmith | mriedem: roger | |
| 15:10:35 | mriedem | stephenfin: can you take a look at https://review.openstack.org/#/c/481116/ and below? | |
| 15:10:52 | mriedem | i'd like to get that fixed and backported through all supported stable branches since it's a regression since newton | |
| 15:11:13 | mriedem | well, less of a regression than a busted feature since newton for evacuating with a target host | |
| 15:12:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: Remove unused block_migration argument to _live_migration_operation https://review.openstack.org/517007 | |
| 15:15:21 | jianghuaw | dansmith, may you take a look at this patch adding an option for enabled_vgpu_type? | |
| 15:15:23 | jianghuaw | https://review.openstack.org/#/c/512580/ | |
| 15:17:33 | dansmith | jianghuaw: I have a short day today and a lot in the queue, but I will try | |
| 15:19:23 | jianghuaw | dansmith, understood. thanks. | |
| 15:19:53 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP Avoid deleting allocations for instances being built https://review.openstack.org/517009 | |
| 15:20:11 | dansmith | mriedem: this is what I'm thinking ^ | |
| 15:21:32 | mriedem | dansmith: yeah i like that | |
| 15:21:47 | mriedem | still maintains the InstanceNotFound logic if the instance is in the cell db and deleted | |
| 15:21:56 | dansmith | we log the situation so if they see the same instance over and over, they know some allocation is stuck | |
| 15:22:20 | dansmith | we _could_ track that set run-to-run and log more forcefully, but that's getting more complex | |
| 15:22:43 | mriedem | https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQ1b-BPsqPMmmI9aPgemul62fGgzoXzhuXWTEde_tBvLbcFaRAA | |
| 15:22:52 | mriedem | always good to know about the situation | |
| 15:23:01 | dansmith | um | |
| 15:24:29 | mriedem | if you want i can handle updating tests and such for that if you're busy today | |
| 15:25:22 | dansmith | I'm working on it now | |
| 15:25:28 | dansmith | I'll punt if I need to | |
| 15:30:08 | mriedem | anthonyper: fyi https://bugs.launchpad.net/nova/+bug/1728924 | |
| 15:30:09 | openstack | Launchpad bug 1728924 in OpenStack Compute (nova) "console logging does not work for OL instances on xen compute" [Undecided,Confirmed] | |
| 15:32:28 | dansmith | mriedem: heh, we have a test that asserts that we abort these unborn children | |
| 15:32:55 | mriedem | i'm not sure how to appropriately respond to that statement | |
| 15:33:17 | dansmith | it asserts the desired behavior for reasons other than "every instance goes through this state" of course | |
| 15:33:26 | dansmith | but it's basically asserting that this buggy behavior exists | |
| 15:33:52 | mriedem | huh | |
| 15:34:00 | mriedem | unrelated, i suppose bauzas is in transit | |
| 15:34:12 | dansmith | already? | |
| 15:34:16 | mriedem | jaypipes: just saw another bug in triage for this same NFS resize bfv issue https://review.openstack.org/#/c/516395/ | |
| 15:34:20 | mriedem | we should get that fixed and backported | |
| 15:34:26 | mriedem | s/fixed/merged/ | |
| 15:34:38 | mriedem | note that https://review.openstack.org/#/c/516396/ verifies it | |
| 15:34:41 | mriedem | melwitt: you too ^ | |