| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-03 | |||
| 09:31:45 | lyarwood | kashyap / stephenfin ; https://review.openstack.org/#/c/579125/ - would you mind taking a swing at this today? | |
| 09:32:36 | kashyap | lyarwood: Oh, this one. Yes, I saw your change. Thanks for addressing | |
| 09:45:03 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add unshelve instance error info to fault table https://review.openstack.org/579747 | |
| 10:22:41 | pooja_jadhav | efried, jaypipes: Actually I want to know the scenario when this method will get called? Ref: https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L3914 | |
| 11:01:35 | hyunsikyang | Hello all | |
| 11:01:35 | hyunsikyang | I am Hyunsik Yang at Soongsil Unviersity, South Korea. | |
| 11:01:35 | hyunsikyang | In the OPNFV project, especially, infra WG, i am trying to provide vBaremetal environment for testing the OPNFV platform. | |
| 11:01:35 | hyunsikyang | For that, IPMI and PXE should be supported well like as a physical server. | |
| 11:01:35 | hyunsikyang | However, in the current OpenStack environment, NOVA manages the lifecycle of the VM, so IPMI operation based on KVM does not work normally. | |
| 11:01:36 | hyunsikyang | To support this, NOVA should support IPMI and PXE. Therefore, this proposal intends to add a function for normal control ofIPMI and PXE in NOVA. | |
| 11:01:36 | hyunsikyang | So, i uploaded spec first. If anyone who interested in this proposal, Please review it. | |
| 11:01:37 | hyunsikyang | Thanks. | |
| 11:01:37 | hyunsikyang | The link is below. | |
| 11:01:38 | hyunsikyang | https://review.openstack.org/#/c/577048/ | |
| 11:03:52 | lyarwood | hyunsikyang: https://docs.openstack.org/tripleo-docs/latest/install/environments/virtualbmc.html - you might want to look at how TripleO already does this with the vbmc project | |
| 11:04:01 | lyarwood | hyunsikyang: https://github.com/openstack/virtualbmc | |
| 11:08:51 | hyunsikyang | I knew it. vBMC which is based on KVM is now working for openstack instance. so, we also make a code for it too. | |
| 11:16:42 | hyunsikyang | lyarwood: I knew it. vBMC which is based on KVM is now working for openstack instance. so, we also make a code for it too. | |
| 11:21:56 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Escalate UUID validation warning to error in test https://review.openstack.org/540386 | |
| 11:22:55 | lyarwood | hyunsikyang: I don't follow, doesn't Ironic provide what you need? | |
| 11:26:48 | hyunsikyang | lyarwood: YES. Ironic used VM Which is based on KVM. | |
| 11:27:26 | hyunsikyang | lyarwood: I want to use openstack instance as a baremetal. | |
| 11:27:44 | lyarwood | hyunsikyang: then just use vbmc to provide the ipmi layer to Ironic? | |
| 11:27:58 | lyarwood | hyunsikyang: nova is never going to directly support ipmi or pxe | |
| 11:31:36 | hyunsikyang | lyarwood: i can't understand what is the meaning of 'then just use vbmc to provide the ipmi layer to Ironic?' | |
| 11:32:18 | hyunsikyang | lyarwood: current nova doesn't support ipmi pxe for openstack instance. | |
| 11:33:25 | lyarwood | hyunsikyang: right, Ironic however does, so the stack is QEMU/KVM instance <-> Nova <-> VBMC <-> Ironic | |
| 11:34:09 | lyarwood | hyunsikyang: VBMC just provides an IPMI layer around the traditional Nova virtual instances that Ironic can use. | |
| 11:41:15 | hyunsikyang | lyarwood: vBMC provides IPMI for VM which is based on KVM. you mean that when user send the command to the vBMC, vBMC will forward this command to the openstack instance? i don't think so. we can control VM which is based on KVM. but we can't controll openstack instance. | |
| 11:42:05 | hyunsikyang | lyarwood: I checked code and tested it. but, it didn't work. | |
| 11:49:04 | lyarwood | hyunsikyang: ah wait, my apologies I mixed up vbmc and ovb - https://openstack-virtual-baremetal.readthedocs.io/en/latest/introduction.html | |
| 11:49:26 | lyarwood | hyunsikyang: vbmc talks to libvirt directly | |
| 11:50:30 | hyunsikyang | Yes I knew it. too:) | |
| 11:50:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Call generate_image_url only for legacy notification https://review.openstack.org/564528 | |
| 11:51:39 | hyunsikyang | but that code also need a modification of nova. nova should support Pxe / IPMI for instance. so i am working on it. | |
| 11:51:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Call generate_image_url only for legacy notification https://review.openstack.org/564528 | |
| 11:52:01 | hyunsikyang | i uploaded two part. one is nova another is vBMC. | |
| 11:52:52 | hyunsikyang | lyarwood: but that code also need a modification of nova. nova should support Pxe / IPMI for instance. so i am working on it. i uploaded two part. one is nova another is vBMC. | |
| 11:53:16 | hyunsikyang | lyarwood: https://review.openstack.org/#/c/570737/ this is for vBMC | |
| 12:19:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enhance service restart in functional env https://review.openstack.org/512552 | |
| 12:19:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1724172 in the functional test env https://review.openstack.org/512553 | |
| 12:19:09 | openstack | bug 1724172 in OpenStack Compute (nova) "Allocation of an evacuated instance is not cleaned on the source host if instance is not defined on the hypervisor" [Medium,In progress] https://launchpad.net/bugs/1724172 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 12:19:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: cleanup evacuated instances not on hypervisor https://review.openstack.org/512623 | |
| 12:49:17 | openstackgerrit | jiang wei proposed openstack/nova master: Add action initiator attribute to the instance payload https://review.openstack.org/536243 | |
| 12:54:23 | openstackgerrit | Merged openstack/nova master: Handle nested serialized json entries in assertJsonEqual https://review.openstack.org/579615 | |
| 13:01:29 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Use ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa in tree https://review.openstack.org/579862 | |
| 13:02:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix unbound local when saving an unchanged RequestSpec https://review.openstack.org/573768 | |
| 13:03:33 | bauzas | mriedem: so, do we have a gate issue ? | |
| 13:03:46 | bauzas | mriedem: I saw your nvm | |
| 13:05:33 | mriedem | you can ignore, it wasn't specific to rescue | |
| 13:05:37 | mriedem | just libvirt randomly crapping out | |
| 13:18:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: cover migration cases with functional tests https://review.openstack.org/493865 | |
| 13:26:48 | efried | pooja_jadhav: I can explain the circumstances under which https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L3914 gets called, but it would help me to know why you're interested in that specific code path. | |
| 13:31:16 | pooja_jadhav | efried: I am eager to learn what DB query is generated when we call method https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/resource_provider.py#L2691. | |
| 13:31:46 | pooja_jadhav | efied: so want to know under which scenarios this method will get called? | |
| 13:37:51 | openstackgerrit | Margarita Shakhova proposed openstack/nova master: Do not skip migrations in _destroy_evacuated_instances() https://review.openstack.org/563623 | |
| 13:42:02 | efried | pooja_jadhav: Okay. If you create a query with a granular request group, you should hit it. | |
| 13:42:33 | efried | pooja_jadhav: Take a look at nova/tests/functional/api/openstack/placement/gabbits/granular.yaml | |
| 13:43:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send soft_delete from context manager https://review.openstack.org/476459 | |
| 13:43:18 | efried | pooja_jadhav: Anywhere you see `resources<Number>` is a query that should hit that code path. | |
| 13:44:57 | pooja_jadhav | efried: ok, will check and if any doubts will let u know. thanks for your time :) | |
| 13:45:21 | efried | Good luck pooja_jadhav | |
| 13:45:35 | jroll | efried: does this ring a bell? sambetts is hitting it in third party CI: http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cpu.txt.gz?level=DEBUG#_Jul_03_12_53_48_638704 | |
| 13:45:49 | jroll | and/or do you have tips on debugging that? or do we need to add more logging | |
| 13:46:01 | efried | jroll: Is it causing a problem? | |
| 13:46:05 | pooja_jadhav | efried: :) | |
| 13:46:15 | efried | jroll: It's actually a semi-expected thing, supposed to self-heal on the next periodic hit. | |
| 13:46:28 | jroll | efried: yes, causes a reschedule (and maybe a failure due to lack of hardware?) | |
| 13:46:47 | jroll | yes, failure | |
| 13:46:49 | jroll | http://192.133.158.227/12/514312/12/experimental/ironic-dsvm-tempest-ironic-cimc-current-centos-7-cimc/f7c455a/logs/screen-n-cond-cell1.txt.gz#_Jul_03_12_53_47_517588 | |
| 13:47:06 | sambetts | yeah, we only have one node, so the retry filter kills the instance | |
| 13:47:13 | jroll | not sure if it's a constant failure or occassional, sam can clarify | |
| 13:47:37 | sambetts | all my CI failures seem to be this at the moment | |
| 13:47:57 | efried | That's weird that it would be affecting a sched operation. At least the one you're showing me ought to be part of periodic, not part of spawn. | |
| 13:48:28 | efried | but in any case, yeah, we have to close some of these gaps in rocky | |
| 13:48:41 | efried | Is this rocky? | |
| 13:48:57 | jroll | oh, that is a periodic, odd | |
| 13:49:50 | efried | What we need to do is put a retry loop around this chunk in the resource tracker: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L883-L914 | |
| 13:50:30 | efried | The devil is in the details, of course. | |
| 13:51:49 | sambetts | looking at the n-cond log that placement sync error is bubbling up in the _build_and_run_instance function | |
| 13:52:37 | sambetts | although its been wrapped in a ReschduledException :/ | |
| 13:52:43 | sambetts | somwhere along the way | |
| 13:52:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Wait for vif plugging during live migration job https://review.openstack.org/578551 | |
| 13:56:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop setting glance_api_version in cinder.conf in nova-live-migration https://review.openstack.org/579871 | |
| 13:57:05 | efried | sambetts: Are you sure it's the same ResourceProviderSyncFailed exception? (Though that's the only one I'm seeing in your compute log.) | |
| 13:59:17 | sambetts | efried: what interesting is that the node UUID in the exeception in n-cpu isn't the node UUID in the n-cond logs | |
| 13:59:48 | sambetts | so I don't know if this is being caused by other nodes async in the background being created and deleted by the API tests | |
| 14:00:16 | sambetts | jroll: ^ | |
| 14:00:37 | efried | sambetts: It isn't? It looks the same to me | |
| 14:00:52 | sambetts | oh acually may bad I had two different cases of this happening up *face plam* | |
| 14:01:14 | jroll | yeah, it is the same :) | |
| 14:01:15 | efried | n-cond: node b5bb2c51-3960-4d9c-b47e-223d83cb6139 | |
| 14:01:15 | efried | n-cpu: Error updating resources for node b5bb2c51-3960-4d9c-b47e-223d83cb6139 | |
| 14:01:23 | efried | okay phew | |
| 14:01:37 | sambetts | I had two jobs for two different ironic drivers and I'm seeing the same error in both so I have like 10 tabs open with all the different logs XD | |
| 14:01:40 | efried | Still mysterious why a failed periodic in n-cpu would cause a spawn to fail. Unless it's not periodic. | |
| 14:01:56 | efried | or unless sambetts is looking at two different sets of logs :P | |
| 14:02:12 | efried | But yeah, I see the resched failure and I'm sure I'm only looking at one :) | |
| 14:02:34 | sambetts | yeah, its definately the same, double checked | |
| 14:04:03 | jroll | hrm, it's the same request id in the reschedule log as the request id in the periodics logs | |