Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-24
12:55:28 bhagyashris efried: As the same test are passing on master and our patches are not making any change at placement or n-sch side
12:56:40 efried bhagyashris: I am also not clear what is needed to fix this, which is why I suggested the above to debug. If we can demonstrate that the inventories in placement are different between master and our patch, then we've done something wrong with the transition from get_inventory to update_provider_tree.
12:56:59 efried bhagyashris: But if they're the same... then we need to look somewhere else (not sure where).
12:57:37 bhagyashris efried: ohh ok
12:58:02 efried bhagyashris: The tests fail the same way with the bottom patch as with the top, right?
12:58:10 efried So the problem isn't with our conditional DISK_GB thing.
12:58:21 efried It's with the translation from get_inventory to update_provider_tree.
12:58:34 efried Could be in the resource tracker.
12:58:39 efried or in the report client.
13:00:15 bhagyashris efried: just checked on both the patches different test are failing
13:01:40 efried bhagyashris: If we're reporting the inventory wrong, it could mean that we're running out of capacity sooner than we used to. But the tempest tests run in parallel in zuul. So which test runs out of capacity first could vary from one run to another.
13:01:56 bhagyashris on parent patch the tempest-full 2 test are failing and on child patch tempest-full 7 tests are failing
13:02:22 efried Let's just focus on the first patch compared to master.
13:03:15 efried bhagyashris: You said you were able to reproduce the failure manually?
13:03:34 bhagyashris efried: no
13:04:37 bhagyashris efried: I have debugged tempest test tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON.test_multiple_create on patch https://review.openstack.org/#/c/560459/ and observed that the allocation_candidates() return at https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L148 is empty list and same test i have ran on master there the allocation_candidates() return is successfully
13:06:10 efried bhagyashris: What do you mean by "debugged" then - you mean just looking at logs?
13:07:53 bhagyashris efried: I mean I have put pdb at https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L121 and ran the tempest test using the tox -e py27 tempest test tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON.test_multiple_create
13:08:20 efried bhagyashris: And you were able to reproduce the fact that GET /allocation_candidates returns no results?
13:08:25 efried ...with our patch
13:08:32 efried ...but on master it returns good results?
13:08:50 bhagyashris efried: yes
13:09:24 efried bhagyashris: Okay, so what I'm asking is for you to do the exact same thing, but when you hit your breakpoint, manually query the placement API and pull back the inventories and allocations for the compute host.
13:10:07 efried bhagyashris: If you do that for both environments - one with our patch and one with master - we should be able to determine whether those inventories/allocations are the same or different.
13:10:39 efried bhagyashris: I'm especially interested in whether the total, reserved, and allocation ratio match; and whether the allocations match.
13:19:22 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Lift the restriction of choices for `cpu_model_extra_flags` https://review.openstack.org/563926
13:19:23 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: Do-Not-Merge: Test multiple CPU flags https://review.openstack.org/563948
13:42:35 openstackgerrit Julia Kreger proposed openstack/nova master: Ironic: Lock the instance if early vif plug https://review.openstack.org/563714
13:51:03 openstackgerrit Jens Harbott (frickler) proposed openstack/nova master: DNM: Test devstack + ceph plugin patches https://review.openstack.org/563097
13:51:14 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix dropped check for boot_index 0 in _validate_bdm https://review.openstack.org/563732
14:01:35 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Improve performance when list instances with IP filter https://review.openstack.org/563972
14:26:32 mriedem stephenfin: can https://review.openstack.org/#/c/541290/ be updated now?
14:27:19 stephenfin mriedem: Sure. I was working through PoC to make sure the idea worked and it does. I'll update the spec pronto
14:27:32 stephenfin *a PoC
14:27:45 mriedem cool, thanks
14:28:28 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Drop MIN_LIBVIRT_BLOCK_LM_WITH_VOLUMES_VERSION https://review.openstack.org/563984
14:33:22 mriedem johnthetubaguy_: can you go over https://review.openstack.org/#/c/552078/ again since i've updated it?
14:37:26 esberglu mriedem: Thoughts on my last comment here? https://review.openstack.org/#/c/526094/45/nova/virt/powervm/driver.py@559
14:38:10 esberglu Ignore the 1st UT comment, that was some confusion on our end
14:38:29 mriedem so it's a DriverVolumeBlockDevice (which is a dict) rather than a BlockDeviceMapping object
14:38:37 mriedem DriverVolumeBlockDevice wraps BlockDeviceMapping
14:38:49 mriedem i think the ironic driver has the same need for the is_volume getting exposed from the internally wrapped bdm
14:38:51 mriedem sec
14:39:26 mriedem https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L392-L396
14:40:08 mriedem so you could do that and then when we handle the ironic driver TODO we can also cleanup the powervm driver
14:40:40 esberglu mriedem: Sounds good thanks!
14:41:00 mriedem that also forces you to use DriverVolumeBlockDevice objects in your unit tests, but i think that's a good thing, given a BDM can be one of at least 3 or 4 things at any given point in the code
14:42:52 esberglu mriedem: We already are using DriverVolumeBlockDevice objects :)
14:43:28 mriedem then you get a root beer scented scratch-n-sniff
14:45:21 stephenfin mriedem, jaypipes, bauzas, gibi: I'm thinking most of the information from here down is too detailed for the spec (it was mostly to help me figure out the problem boundary myself). I think I could drop the whole piece. Thoughts? https://review.openstack.org/#/c/541290/7/specs/rocky/approved/numa-aware-vswitches.rst@38
14:47:02 mriedem stephenfin: given my lack of knowledge on numa stuff, i'll likely appreciate details in the problem description
14:48:00 stephenfin mriedem: This is more about OVS-DPDK internals. If you think that's be helpful, I can keep it
14:48:12 stephenfin I'd like to add it to a different section but that upsets pep8 :)
14:48:29 gibi stephenfin: I have the same mental debate about the bandwidth spec. It is too detailed for a reader who is familar with the problem and the proposed solution, but it has a lot of nice details and reasoning for a new reader.
14:48:44 gibi stephenfin: for me it is OK to remove that section from your spec
14:49:09 gibi stephenfin: you can add new subsections but you cannot add a new top level section
14:49:11 stephenfin maybe I can publish it as a separate blog and link to that from there
14:49:23 mriedem stephenfin: i was just thinking that
14:49:50 mriedem 'for more information on how this relates to dpdk, see $link'
14:50:02 stephenfin mriedem: Sounds good to me. I'll do that
14:53:55 jaypipes stephenfin: I don't mind that detail.
15:05:54 kashyap stephenfin: Yeah, it comes useful for that poor soul who will look at it 5 years down the line
15:14:03 melwitt lyaaaaaaaaarwood: could you please hit this again? pike change has merged https://review.openstack.org/#/c/561613/
15:15:27 melwitt dansmith: could you please look at this stack of two backports for ocata? these and the one ^ are the last needed for the ocata release https://review.openstack.org/#/c/560162
15:17:39 dansmith ack
15:19:37 openstackgerrit Jay Pipes proposed openstack/nova-specs master: Support initial allocation ratios https://review.openstack.org/552105
15:19:57 lyarwood melwitt: done
15:20:17 melwitt mriedem: what do you think of this approach for fixing the ceph job? frickler is trying something different to check the target branch to determine "if pike uca" https://review.openstack.org/#/c/563870
15:20:55 melwitt thanks lyarwood
15:23:08 mriedem melwitt: since stable/pike devstack uses the ocata UCA, and rocky now uses the queens UCA, and devstack-plugin-ceph is branchless, this seems appropriate
15:24:36 melwitt mriedem: k, cool. just wanted to make sure stable/queens won't be changing which UCA it uses in the future (makes sense that it wouldn't change)
15:24:56 mriedem it could change, but likely wont
15:24:58 kashyap Does anyone with Parallels / Virtuozzo experience, do you know if QEMU Guest Agent is required to set password in Nova?
15:25:18 mriedem kashyap: you'd have to reach out to mnestratov
15:25:42 kashyap mriedem: Yep, checking with one of his colleagues
15:25:45 mriedem https://wiki.openstack.org/wiki/ThirdPartySystems/Virtuozzo_CI
15:25:47 melwitt cool
15:25:55 kashyap As Maxim normally doesn't seem to hangout here, only occasionally
15:28:16 kashyap None of the contacts are on IRC (neither on FN, nor on OFTC), I'll email them probably
15:37:24 kashyap Sent
15:38:06 openstackgerrit Merged openstack/os-vif master: Trivial: Update pypi url to new url https://review.openstack.org/563246
15:39:56 openstackgerrit Jay Pipes proposed openstack/nova master: mirror nova host aggregate members to placement https://review.openstack.org/553597
15:40:52 TheJulia Greetings nova folk, we're currently looking at an issue with ironic virt driver where due to the need for networking information for block device mappings, we end up getting called for vif attachment actions prior to a node being reserved in the spawn action by our virt driver. We're pondering two options, explicitly check during a vif plugging action, or adding a new virt driver call that would be along the lines of
15:40:52 TheJulia preparing to spawn in order to facilitate the actual lock of the node resource. Thoughts would be appreciated since we somehow need to flag the node as in use for any other users of ironic's API prior to attaching the vifs.
15:41:12 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI Fibre Channel volume adapter https://review.openstack.org/526094
15:41:13 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: Snapshot https://review.openstack.org/543023
15:41:14 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: DiskAdapter parent class https://review.openstack.org/549053
15:41:16 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300
15:42:38 melwitt jbernard: hi, we're trying to fix the ceph job that's failing 100% on master, would appreciate your review https://review.openstack.org/#/c/563870
15:43:04 jbernard melwitt: certainly
15:43:12 melwitt thanks!
15:43:38 jaypipes TheJulia: I'm confused why vif setup actions are being called prior to the node being reserved.
15:44:12 melwitt same
15:44:27 jaypipes TheJulia: I would think that setup_networking_on_host() would only happen after Ironic has notified the Ironic virt driver that the node is ready for provisioining?
15:47:27 TheJulia jaypipes: melwitt: let me grab the link for the change so we can discuss this with more information
15:47:39 TheJulia between three other conversations :(
15:48:19 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Drop MIN_LIBVIRT_BLOCK_LM_WITH_VOLUMES_VERSION https://review.openstack.org/563984
15:48:20 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Drop MIN_LIBVIRT_NUMA_VERSION_PPC https://review.openstack.org/564010
15:48:21 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: Drop BAD_LIBVIRT_NUMA_VERSIONS https://review.openstack.org/564011
15:48:22 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Drop BAD_LIBVIRT_CPU_POLICY_VERSIONS https://review.openstack.org/564012

Earlier   Later