| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-03 | |||
| 11:42:05 | hyunsikyang | lyarwood: I checked code and tested it. but, it didn't work. | |
| 11:49:04 | lyarwood | hyunsikyang: ah wait, my apologies I mixed up vbmc and ovb - https://openstack-virtual-baremetal.readthedocs.io/en/latest/introduction.html | |
| 11:49:26 | lyarwood | hyunsikyang: vbmc talks to libvirt directly | |
| 11:50:30 | hyunsikyang | Yes I knew it. too:) | |
| 11:50:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Call generate_image_url only for legacy notification https://review.openstack.org/564528 | |
| 11:51:39 | hyunsikyang | but that code also need a modification of nova. nova should support Pxe / IPMI for instance. so i am working on it. | |
| 11:51:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Call generate_image_url only for legacy notification https://review.openstack.org/564528 | |
| 11:52:01 | hyunsikyang | i uploaded two part. one is nova another is vBMC. | |
| 11:52:52 | hyunsikyang | lyarwood: but that code also need a modification of nova. nova should support Pxe / IPMI for instance. so i am working on it. i uploaded two part. one is nova another is vBMC. | |
| 11:53:16 | hyunsikyang | lyarwood: https://review.openstack.org/#/c/570737/ this is for vBMC | |
| 12:19:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enhance service restart in functional env https://review.openstack.org/512552 | |
| 12:19:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1724172 in the functional test env https://review.openstack.org/512553 | |
| 12:19:09 | openstack | bug 1724172 in OpenStack Compute (nova) "Allocation of an evacuated instance is not cleaned on the source host if instance is not defined on the hypervisor" [Medium,In progress] https://launchpad.net/bugs/1724172 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 12:19:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: cleanup evacuated instances not on hypervisor https://review.openstack.org/512623 | |
| 12:49:17 | openstackgerrit | jiang wei proposed openstack/nova master: Add action initiator attribute to the instance payload https://review.openstack.org/536243 | |
| 12:54:23 | openstackgerrit | Merged openstack/nova master: Handle nested serialized json entries in assertJsonEqual https://review.openstack.org/579615 | |
| 13:01:29 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Use ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa in tree https://review.openstack.org/579862 | |
| 13:02:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix unbound local when saving an unchanged RequestSpec https://review.openstack.org/573768 | |
| 13:03:33 | bauzas | mriedem: so, do we have a gate issue ? | |
| 13:03:46 | bauzas | mriedem: I saw your nvm | |
| 13:05:33 | mriedem | you can ignore, it wasn't specific to rescue | |
| 13:05:37 | mriedem | just libvirt randomly crapping out | |
| 13:18:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: cover migration cases with functional tests https://review.openstack.org/493865 | |
| 13:26:48 | efried | pooja_jadhav: I can explain the circumstances under which https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L3914 gets called, but it would help me to know why you're interested in that specific code path. | |
| 13:31:16 | pooja_jadhav | efried: I am eager to learn what DB query is generated when we call method https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L2691. | |
| 13:31:46 | pooja_jadhav | efied: so want to know under which scenarios this method will get called? | |
| 13:37:51 | openstackgerrit | Margarita Shakhova proposed openstack/nova master: Do not skip migrations in _destroy_evacuated_instances() https://review.openstack.org/563623 | |
| 13:42:02 | efried | pooja_jadhav: Okay. If you create a query with a granular request group, you should hit it. | |
| 13:42:33 | efried | pooja_jadhav: Take a look at nova/tests/functional/api/openstack/placement/gabbits/granular.yaml | |
| 13:43:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send soft_delete from context manager https://review.openstack.org/476459 | |
| 13:43:18 | efried | pooja_jadhav: Anywhere you see `resources<Number>` is a query that should hit that code path. | |
| 13:44:57 | pooja_jadhav | efried: ok, will check and if any doubts will let u know. thanks for your time :) | |
| 13:45:21 | efried | Good luck pooja_jadhav | |
| 13:45:35 | jroll | efried: does this ring a bell? sambetts is hitting it in third party CI: http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jul_03_12_53_48_638704 | |
| 13:45:49 | jroll | and/or do you have tips on debugging that? or do we need to add more logging | |
| 13:46:01 | efried | jroll: Is it causing a problem? | |
| 13:46:05 | pooja_jadhav | efried: :) | |
| 13:46:15 | efried | jroll: It's actually a semi-expected thing, supposed to self-heal on the next periodic hit. | |
| 13:46:28 | jroll | efried: yes, causes a reschedule (and maybe a failure due to lack of hardware?) | |
| 13:46:47 | jroll | yes, failure | |
| 13:46:49 | jroll | http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cond-cell1.txt.gz#_Jul_03_12_53_47_517588 | |
| 13:47:06 | sambetts | yeah, we only have one node, so the retry filter kills the instance | |
| 13:47:13 | jroll | not sure if it's a constant failure or occassional, sam can clarify | |
| 13:47:37 | sambetts | all my CI failures seem to be this at the moment | |
| 13:47:57 | efried | That's weird that it would be affecting a sched operation. At least the one you're showing me ought to be part of periodic, not part of spawn. | |
| 13:48:28 | efried | but in any case, yeah, we have to close some of these gaps in rocky | |
| 13:48:41 | efried | Is this rocky? | |
| 13:48:57 | jroll | oh, that is a periodic, odd | |
| 13:49:50 | efried | What we need to do is put a retry loop around this chunk in the resource tracker: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L883-L914 | |
| 13:50:30 | efried | The devil is in the details, of course. | |
| 13:51:49 | sambetts | looking at the n-cond log that placement sync error is bubbling up in the _build_and_run_instance function | |
| 13:52:37 | sambetts | although its been wrapped in a ReschduledException :/ | |
| 13:52:43 | sambetts | somwhere along the way | |
| 13:52:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Wait for vif plugging during live migration job https://review.openstack.org/578551 | |
| 13:56:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop setting glance_api_version in cinder.conf in nova-live-migration https://review.openstack.org/579871 | |
| 13:57:05 | efried | sambetts: Are you sure it's the same ResourceProviderSyncFailed exception? (Though that's the only one I'm seeing in your compute log.) | |
| 13:59:17 | sambetts | efried: what interesting is that the node UUID in the exeception in n-cpu isn't the node UUID in the n-cond logs | |
| 13:59:48 | sambetts | so I don't know if this is being caused by other nodes async in the background being created and deleted by the API tests | |
| 14:00:16 | sambetts | jroll: ^ | |
| 14:00:37 | efried | sambetts: It isn't? It looks the same to me | |
| 14:00:52 | sambetts | oh acually may bad I had two different cases of this happening up *face plam* | |
| 14:01:14 | jroll | yeah, it is the same :) | |
| 14:01:15 | efried | n-cond: node b5bb2c51-3960-4d9c-b47e-223d83cb6139 | |
| 14:01:15 | efried | n-cpu: Error updating resources for node b5bb2c51-3960-4d9c-b47e-223d83cb6139 | |
| 14:01:23 | efried | okay phew | |
| 14:01:37 | sambetts | I had two jobs for two different ironic drivers and I'm seeing the same error in both so I have like 10 tabs open with all the different logs XD | |
| 14:01:40 | efried | Still mysterious why a failed periodic in n-cpu would cause a spawn to fail. Unless it's not periodic. | |
| 14:01:56 | efried | or unless sambetts is looking at two different sets of logs :P | |
| 14:02:12 | efried | But yeah, I see the resched failure and I'm sure I'm only looking at one :) | |
| 14:02:34 | sambetts | yeah, its definately the same, double checked | |
| 14:04:03 | jroll | hrm, it's the same request id in the reschedule log as the request id in the periodics logs | |
| 14:05:59 | sambetts | the trace isn't quite deep enough in the n-cpu logs to know what called _update_available_resource_for_node | |
| 14:06:51 | sambetts | only place I can see it being called is here: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7479 | |
| 14:06:54 | jroll | sambetts: it's only from the periodic task: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7500 | |
| 14:07:16 | jroll | odd | |
| 14:08:34 | efried | I'm pretty sure there's a code path that hits that guy during spawn. | |
| 14:09:04 | efried | I don't remember where in the stack it happens. We're, what, five layers deep in methods called update-something? | |
| 14:09:35 | efried | but I would have expected to see it in this stack trace. | |
| 14:09:49 | jroll | efried: calls which guy, specifically? | |
| 14:10:00 | efried | that's what I don't remember. | |
| 14:10:27 | efried | jroll: I'm saying I think there's a code path during spawn that winds up here in _update, but I don't remember where in the call stack it's initiated. | |
| 14:10:37 | efried | that didn't make sense, gimme a sec to find what I'm talking about. | |
| 14:10:48 | jroll | hm, ok | |
| 14:11:06 | jroll | the compute traceback is pretty clearly from the periodic task | |
| 14:11:15 | jroll | per http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jul_03_12_53_48_624944 | |
| 14:11:35 | jroll | er wait, that's later | |
| 14:11:49 | jroll | ignore me | |
| 14:12:01 | jroll | (also, I see what you're saying) | |
| 14:13:26 | sambetts | its a same that because a new exepcetion is raised in the n-cond code the line that through the original exeception is lost | |
| 14:14:27 | sambetts | s/thought/threw | |
| 14:18:30 | efried | I am now unable to find the code path that calls this from not-a-periodic. | |
| 14:18:46 | jroll | instance_claim() calls into _update() | |
| 14:19:17 | jroll | which is in _build_and_run_instance | |
| 14:19:29 | jroll | so that's probably where the build is bombing | |
| 14:19:42 | jroll | and the n-cpu traceback is an unrelated red herring | |
| 14:20:18 | sambetts | same exception on both sides though :/ | |
| 14:20:33 | efried | jroll: Then why is that exception message in the resched... yeah | |
| 14:20:34 | jroll | yes | |
| 14:20:45 | jroll | same exception, different call stack | |
| 14:21:00 | sambetts | maybe the reason they are both raising that execetion is because of a collision? | |