Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-15
16:25:51 ttsiouts Hi! Anyone interested in reviewing this? https://review.openstack.org/#/c/525110/6 I would really appreciate the help!
16:25:52 figleaf mriedem: any further thoughts on your comment here: https://review.openstack.org/#/c/511358/38/nova/conductor/manager.py@568
16:26:00 mriedem lyarwood: artom: heh, well -2
16:26:02 lyarwood artom: \o/ np
16:26:15 figleaf mriedem: I can't figure out how that's getting hit
16:26:16 mriedem figleaf: requires tracing the request id through the logs in that ironic job
16:26:21 mriedem figleaf: i just haven't had the time to do that yet
16:26:33 mriedem we should at least figure out if we are getting the alternates correctly in that CI run
16:26:50 ttsiouts mriedem: can I ask you about the --force parameter in nova quota-update?
16:26:56 figleaf maybe add some debug logging?
16:27:01 mriedem so first question is, do we get alternates at all? second is, are we using them properly? and if so, why aren't we quitting the retry loop properly?
16:27:10 lyarwood mriedem: so this is never getting into Newton or are we still going to wait to see if things break in master?
16:27:16 mriedem figleaf: sure
16:27:41 mriedem lyarwood: i think by the time we'd find out if this regresses something in master, or pike or ocata, we'll have wanted to eol newton already
16:27:57 mriedem which is why i don't want to hold newton for that change, and i think it's risky to put it in before eol
16:28:03 lyarwood ack
16:28:14 mriedem we can, and should, however, backport the schema change
16:28:15 mriedem imo
16:28:26 mriedem ttsiouts: sure
16:28:36 ttsiouts mriedem: thanks
16:28:37 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Enable limiting GET /allocation_candidates https://review.openstack.org/513526
16:28:51 mriedem figleaf: yeah maybe log the primary host and alternates we get in schedule_and_build_instances,
16:28:59 ttsiouts mriedem: it seems that it is missing from the openstackclient
16:29:03 mriedem and then log the chosen alternate in build_instances before casting to the alternate compute
16:29:06 artom lyarwood, right, let's actually speak to each other on this one: https://review.openstack.org/#/c/528012/ your backport or mine?
16:29:21 mriedem artom: you do the backports
16:29:24 mriedem so lee and i can approve them
16:29:28 lyarwood artom: yup
16:29:34 figleaf mriedem: I can add a patch below my series that just adds that debugging
16:29:36 ttsiouts mriedem: cinder and neutron have no problem setting the quota to a value less than the used..
16:30:09 mriedem figleaf: ok - there is actually a patch for logging the host at the top, sec
16:30:29 mriedem figleaf: https://review.openstack.org/#/c/520369/
16:30:34 mriedem maybe rebase on top of that?
16:31:00 figleaf mriedem: sounds good
16:31:02 mriedem figleaf: another thing this might be,
16:31:11 mriedem not sure, but the host:node cardinality with ironic
16:31:28 mriedem i wonder if we don't have alternates because hosts < nodes
16:31:40 mriedem but, this is a 2 host job, so we should at least have one alternate i'd think?
16:31:52 figleaf 2 host or 2 node?
16:32:01 mriedem the multinode job should be 2 compute services
16:32:02 mriedem so 2 hosts
16:32:05 figleaf ok
16:32:07 mriedem i don't know how many nodes it has per host
16:32:37 mriedem but, i think this also means that the alternates stuff doesn't necessarily solve the reschedule problem for ironic, not in it's current form anyway
16:32:50 openstackgerrit Andrey Kurilin proposed openstack/python-novaclient master: Remove 2 redundant methods https://review.openstack.org/528325
16:33:01 mriedem because we could have host:nodeA and host:nodeB, fail on nodeA and not try to reschedule to nodeB
16:33:15 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Verify parameters in servers.inc https://review.openstack.org/528201
16:33:16 mriedem because we think, "oh well that host failed so i'm not going to try it again"
16:33:22 figleaf mriedem: yeah, I remember having that discussion with some ironic people in Denver
16:33:28 mriedem just a guess at this point
16:33:54 mriedem i think we'd know if we have debug logs that show we try host 1, reschedule to host 2, and then need to reschedule again and are out of hosts but not out of nodes
16:34:10 andreykurilin_ mriedem: and a simple cleanup for you - https://review.openstack.org/528325 :)
16:35:02 mriedem ttsiouts: so your issue is that osc doesn't have a --force option for nova quota-update?
16:35:11 artom Wait, we don't have pike placeholders?
16:35:19 mriedem ttsiouts: if so, feel free to write a patch to openstackclient
16:35:41 artom All I see is Dan's a5e051edd9 adding the Ocata ones
16:35:56 ttsiouts mriedem: thanks
16:36:56 artom Ah, no, I was confused
16:37:14 artom The Pike placeholders will be in master, so it's b198ad349a that added them
16:37:21 artom The Ocata placeholder in Pike, etc
16:37:33 mriedem yes
16:37:39 mriedem https://review.openstack.org/#/c/427900/ was done in pike for the ocata backports
16:37:41 mriedem whew
16:40:59 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330
16:43:27 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
16:43:39 melwitt lyarwood: these stable backports would appreciate your review https://review.openstack.org/#/c/523910 https://review.openstack.org/#/c/523911
16:43:56 openstackgerrit Artom Lifshitz proposed openstack/nova stable/ocata: Make request_spec.spec MediumText https://review.openstack.org/528332
16:43:58 figleaf mriedem: ugh, can't log the alternates in build_instances on that patch, since that method doesn't get alternates until https://review.openstack.org/#/c/511358
16:44:21 figleaf so I don't think that alternates could be causing the ironic failure
16:44:24 mriedem figleaf: i think it's fine to just update your patches to log the alternates at debug while we debug the actual issue
16:44:36 mriedem figleaf: well the ironic patch is trying to reschedule
16:44:38 figleaf since they aren't yet being passed
16:44:47 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330
16:44:52 mriedem the ironic patch is dependent on the patch that passes the alternates to compute
16:45:21 figleaf ok, didn't know it was using that change
16:45:40 figleaf I'll add the logging on top of that
16:49:36 openstackgerrit Artom Lifshitz proposed openstack/nova stable/newton: Make request_spec.spec MediumText https://review.openstack.org/528338
17:10:34 artom So, do we really attempt a live migration, if requested, even with SRIOV stuff attached, when we know it'll fail, and we just let it fail instead of failing fast at a higher level?
17:14:05 claudiub i wouldn't want that, imo.
17:14:23 claudiub live migration with SRIOV should still work for hyper-v.
17:14:47 artom Ah, indeed
17:15:01 artom So libvirt should fail properly, at least
17:15:10 openstackgerrit Merged openstack/nova stable/pike: Pass the correct image to build_request_spec in conductor.rebuild_instance https://review.openstack.org/516400
17:15:18 openstackgerrit Merged openstack/nova master: SchedulerReportClient._get_providers_in_aggregates https://review.openstack.org/521097
17:15:27 openstackgerrit Merged openstack/nova stable/ocata: Add release note for running nova-api under wsgi in Ocata https://review.openstack.org/511503
17:15:29 claudiub indeed.
17:16:03 claudiub and afaik, as long as you find a proper host for live migration, which has the exact PF / VF available, it should still be live migratable
17:19:09 artom claudiub, all of the docs state that live migration with SRIOV and libvirt/kvm isn't supported
17:19:12 artom So I know it doesn't work
17:19:20 artom I'm trying to find out how much it blows up
17:20:14 claudiub hm, well, if something is not supported, it doesn't mean it isn't working at all, or at least that's what I understand. :)
17:21:06 cdent claudiub++
18:08:59 openstackgerrit Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363
18:08:59 openstackgerrit Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362
18:30:04 openstackgerrit Merged openstack/nova stable/ocata: Document the real behavior of notify_on_state_change https://review.openstack.org/521871
18:40:10 openstackgerrit Merged openstack/nova stable/pike: Fix TypeError in nova-manage db archive_deleted_rows https://review.openstack.org/525629
18:41:09 openstackgerrit Merged openstack/nova master: Make request_spec.spec MediumText https://review.openstack.org/528012
18:41:16 openstackgerrit Merged openstack/nova master: [placement] Separate API schemas (usage) https://review.openstack.org/520603
18:41:23 openstackgerrit Merged openstack/nova master: [placement] Separate API schemas (aggregate) https://review.openstack.org/520608
19:48:37 openstackgerrit Ade Lee proposed openstack/nova master: Add flag to make glance image verification optional https://review.openstack.org/528017

Earlier   Later