| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-15 | |||
| 16:26:02 | lyarwood | artom: \o/ np | |
| 16:26:15 | figleaf | mriedem: I can't figure out how that's getting hit | |
| 16:26:16 | mriedem | figleaf: requires tracing the request id through the logs in that ironic job | |
| 16:26:21 | mriedem | figleaf: i just haven't had the time to do that yet | |
| 16:26:33 | mriedem | we should at least figure out if we are getting the alternates correctly in that CI run | |
| 16:26:50 | ttsiouts | mriedem: can I ask you about the --force parameter in nova quota-update? | |
| 16:26:56 | figleaf | maybe add some debug logging? | |
| 16:27:01 | mriedem | so first question is, do we get alternates at all? second is, are we using them properly? and if so, why aren't we quitting the retry loop properly? | |
| 16:27:10 | lyarwood | mriedem: so this is never getting into Newton or are we still going to wait to see if things break in master? | |
| 16:27:16 | mriedem | figleaf: sure | |
| 16:27:41 | mriedem | lyarwood: i think by the time we'd find out if this regresses something in master, or pike or ocata, we'll have wanted to eol newton already | |
| 16:27:57 | mriedem | which is why i don't want to hold newton for that change, and i think it's risky to put it in before eol | |
| 16:28:03 | lyarwood | ack | |
| 16:28:14 | mriedem | we can, and should, however, backport the schema change | |
| 16:28:15 | mriedem | imo | |
| 16:28:26 | mriedem | ttsiouts: sure | |
| 16:28:36 | ttsiouts | mriedem: thanks | |
| 16:28:37 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Enable limiting GET /allocation_candidates https://review.openstack.org/513526 | |
| 16:28:51 | mriedem | figleaf: yeah maybe log the primary host and alternates we get in schedule_and_build_instances, | |
| 16:28:59 | ttsiouts | mriedem: it seems that it is missing from the openstackclient | |
| 16:29:03 | mriedem | and then log the chosen alternate in build_instances before casting to the alternate compute | |
| 16:29:06 | artom | lyarwood, right, let's actually speak to each other on this one: https://review.openstack.org/#/c/528012/ your backport or mine? | |
| 16:29:21 | mriedem | artom: you do the backports | |
| 16:29:24 | mriedem | so lee and i can approve them | |
| 16:29:28 | lyarwood | artom: yup | |
| 16:29:34 | figleaf | mriedem: I can add a patch below my series that just adds that debugging | |
| 16:29:36 | ttsiouts | mriedem: cinder and neutron have no problem setting the quota to a value less than the used.. | |
| 16:30:09 | mriedem | figleaf: ok - there is actually a patch for logging the host at the top, sec | |
| 16:30:29 | mriedem | figleaf: https://review.openstack.org/#/c/520369/ | |
| 16:30:34 | mriedem | maybe rebase on top of that? | |
| 16:31:00 | figleaf | mriedem: sounds good | |
| 16:31:02 | mriedem | figleaf: another thing this might be, | |
| 16:31:11 | mriedem | not sure, but the host:node cardinality with ironic | |
| 16:31:28 | mriedem | i wonder if we don't have alternates because hosts < nodes | |
| 16:31:40 | mriedem | but, this is a 2 host job, so we should at least have one alternate i'd think? | |
| 16:31:52 | figleaf | 2 host or 2 node? | |
| 16:32:01 | mriedem | the multinode job should be 2 compute services | |
| 16:32:02 | mriedem | so 2 hosts | |
| 16:32:05 | figleaf | ok | |
| 16:32:07 | mriedem | i don't know how many nodes it has per host | |
| 16:32:37 | mriedem | but, i think this also means that the alternates stuff doesn't necessarily solve the reschedule problem for ironic, not in it's current form anyway | |
| 16:32:50 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient master: Remove 2 redundant methods https://review.openstack.org/528325 | |
| 16:33:01 | mriedem | because we could have host:nodeA and host:nodeB, fail on nodeA and not try to reschedule to nodeB | |
| 16:33:15 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Verify parameters in servers.inc https://review.openstack.org/528201 | |
| 16:33:16 | mriedem | because we think, "oh well that host failed so i'm not going to try it again" | |
| 16:33:22 | figleaf | mriedem: yeah, I remember having that discussion with some ironic people in Denver | |
| 16:33:28 | mriedem | just a guess at this point | |
| 16:33:54 | mriedem | i think we'd know if we have debug logs that show we try host 1, reschedule to host 2, and then need to reschedule again and are out of hosts but not out of nodes | |
| 16:34:10 | andreykurilin_ | mriedem: and a simple cleanup for you - https://review.openstack.org/528325 :) | |
| 16:35:02 | mriedem | ttsiouts: so your issue is that osc doesn't have a --force option for nova quota-update? | |
| 16:35:11 | artom | Wait, we don't have pike placeholders? | |
| 16:35:19 | mriedem | ttsiouts: if so, feel free to write a patch to openstackclient | |
| 16:35:41 | artom | All I see is Dan's a5e051edd9 adding the Ocata ones | |
| 16:35:56 | ttsiouts | mriedem: thanks | |
| 16:36:56 | artom | Ah, no, I was confused | |
| 16:37:14 | artom | The Pike placeholders will be in master, so it's b198ad349a that added them | |
| 16:37:21 | artom | The Ocata placeholder in Pike, etc | |
| 16:37:33 | mriedem | yes | |
| 16:37:39 | mriedem | https://review.openstack.org/#/c/427900/ was done in pike for the ocata backports | |
| 16:37:41 | mriedem | whew | |
| 16:40:59 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330 | |
| 16:43:27 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 16:43:39 | melwitt | lyarwood: these stable backports would appreciate your review https://review.openstack.org/#/c/523910 https://review.openstack.org/#/c/523911 | |
| 16:43:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/ocata: Make request_spec.spec MediumText https://review.openstack.org/528332 | |
| 16:43:58 | figleaf | mriedem: ugh, can't log the alternates in build_instances on that patch, since that method doesn't get alternates until https://review.openstack.org/#/c/511358 | |
| 16:44:21 | figleaf | so I don't think that alternates could be causing the ironic failure | |
| 16:44:24 | mriedem | figleaf: i think it's fine to just update your patches to log the alternates at debug while we debug the actual issue | |
| 16:44:36 | mriedem | figleaf: well the ironic patch is trying to reschedule | |
| 16:44:38 | figleaf | since they aren't yet being passed | |
| 16:44:47 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330 | |
| 16:44:52 | mriedem | the ironic patch is dependent on the patch that passes the alternates to compute | |
| 16:45:21 | figleaf | ok, didn't know it was using that change | |
| 16:45:40 | figleaf | I'll add the logging on top of that | |
| 16:49:36 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/newton: Make request_spec.spec MediumText https://review.openstack.org/528338 | |
| 17:10:34 | artom | So, do we really attempt a live migration, if requested, even with SRIOV stuff attached, when we know it'll fail, and we just let it fail instead of failing fast at a higher level? | |
| 17:14:05 | claudiub | i wouldn't want that, imo. | |
| 17:14:23 | claudiub | live migration with SRIOV should still work for hyper-v. | |
| 17:14:47 | artom | Ah, indeed | |
| 17:15:01 | artom | So libvirt should fail properly, at least | |
| 17:15:10 | openstackgerrit | Merged openstack/nova stable/pike: Pass the correct image to build_request_spec in conductor.rebuild_instance https://review.openstack.org/516400 | |
| 17:15:18 | openstackgerrit | Merged openstack/nova master: SchedulerReportClient._get_providers_in_aggregates https://review.openstack.org/521097 | |
| 17:15:27 | openstackgerrit | Merged openstack/nova stable/ocata: Add release note for running nova-api under wsgi in Ocata https://review.openstack.org/511503 | |
| 17:15:29 | claudiub | indeed. | |
| 17:16:03 | claudiub | and afaik, as long as you find a proper host for live migration, which has the exact PF / VF available, it should still be live migratable | |
| 17:19:09 | artom | claudiub, all of the docs state that live migration with SRIOV and libvirt/kvm isn't supported | |
| 17:19:12 | artom | So I know it doesn't work | |
| 17:19:20 | artom | I'm trying to find out how much it blows up | |
| 17:20:14 | claudiub | hm, well, if something is not supported, it doesn't mean it isn't working at all, or at least that's what I understand. :) | |
| 17:21:06 | cdent | claudiub++ | |
| 18:08:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | |
| 18:08:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | |
| 18:30:04 | openstackgerrit | Merged openstack/nova stable/ocata: Document the real behavior of notify_on_state_change https://review.openstack.org/521871 | |
| 18:40:10 | openstackgerrit | Merged openstack/nova stable/pike: Fix TypeError in nova-manage db archive_deleted_rows https://review.openstack.org/525629 | |
| 18:41:09 | openstackgerrit | Merged openstack/nova master: Make request_spec.spec MediumText https://review.openstack.org/528012 | |
| 18:41:16 | openstackgerrit | Merged openstack/nova master: [placement] Separate API schemas (usage) https://review.openstack.org/520603 | |
| 18:41:23 | openstackgerrit | Merged openstack/nova master: [placement] Separate API schemas (aggregate) https://review.openstack.org/520608 | |
| 19:48:37 | openstackgerrit | Ade Lee proposed openstack/nova master: Add flag to make glance image verification optional https://review.openstack.org/528017 | |
| 19:48:54 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 19:48:55 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 19:49:16 | figleaf | mriedem: ^^ Added logging | |