| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-26 | |||
| 06:52:59 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Microversion 2.74: Support Specifying AZ to unshelve https://review.opendev.org/665136 | |
| 07:29:24 | brinzhang | efried: Are you around? | |
| 08:25:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: pull out functions from _heal_allocations_for_instance https://review.opendev.org/655457 | |
| 08:25:32 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prepare _heal_allocations_for_instance for nested allocations https://review.opendev.org/637954 | |
| 08:25:32 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: reorder conditions in _heal_allocations_for_instance https://review.opendev.org/655458 | |
| 08:25:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.opendev.org/637955 | |
| 08:25:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: pull out put_allocation call from _heal_* https://review.opendev.org/655459 | |
| 09:04:09 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Grab fresh power state info from the driver https://review.opendev.org/665975 | |
| 09:29:12 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Update AZ admin doc to mention the new way to specify hosts https://review.opendev.org/666767 | |
| 09:32:45 | kashyap | Does anyone here of an existing bug in the Gate, where the "tempest-slow-py3" is failing with: | |
| 09:32:48 | kashyap | tempest.exceptions.BuildErrorException: Server 008c5c50-ff54-49f4-adb0-23775e8af5f1 failed to build and is in ERROR status | |
| 09:32:51 | kashyap | Details: {'code': 500, 'created': '2019-06-25T20:55:49Z', 'message': 'Unexpected vif_type=binding_failed'} | |
| 09:32:54 | kashyap | http://logs.openstack.org/89/667389/1/check/tempest-slow-py3/2606bcc/testr_results.html.gz | |
| 09:32:59 | kashyap | [That is a stable/stein backport] | |
| 09:37:21 | kashyap | Okay, I see time outs (also in the stable/rocky backport) in the 'testr_results'. /me goes to 'recheck' | |
| 11:06:46 | NewBruce | Hey @kayshap | |
| 11:07:47 | NewBruce | so good news, i didnt try to mess around with xml, instead just use SELinux ;) but not sure if you can help out on this one - migrations are failing with | |
| 11:07:48 | NewBruce | Live Migration failure: Library function returned error but did not set virError: libvirtError: Library function returned error but did not set vir | |
| 11:08:17 | NewBruce | 2019-06-26T09:46:22.815036Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-net' | |
| 11:08:17 | NewBruce | 2019-06-26T09:46:22.815033Z qemu-kvm: Failed to load virtio-net:virtio | |
| 11:08:17 | NewBruce | 2019-06-26 09:46:23.190+0000: 19228: error : virNetClientProgramDispatchError:177 : internal error: qemu unexpectedly closed the monitor: 2019-06-26T09:46:22.815029Z qemu-kvm: Failed to load PCIDevice:config | |
| 11:08:17 | NewBruce | 2019-06-26 09:46:22.816+0000: 30621: error : virNetClientStreamRaiseError:200 : stream had I/O failure | |
| 11:08:17 | NewBruce | digging into the libvirt logs - | |
| 11:09:34 | NewBruce | thing is, from the control side of things, the migration completed - no errors or anything are returned… also, i could migrate fine between these hosts before i added SELinux, and it (rarely) works to migrate a machine… im lost if its a LibVirt or nova issue at this point - thoughts? | |
| 11:10:12 | NewBruce | Progress: 97Complete | |
| 11:10:12 | NewBruce | the osc reports life is peachy : openstack server migrate --live cc-compute28-sto2 aadfe56a-88b8-49c0-9dac-41a4c494c1b5 --wait | |
| 11:11:25 | NewBruce | Migration operation has aborted | |
| 11:11:25 | NewBruce | Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) | |
| 11:11:25 | NewBruce | Took 2.35 seconds for pre_live_migration on destination host cc-compute26-sto2. | |
| 11:11:25 | NewBruce | but nova never gets to post-migration, and i dont think its actually doing the migration itself - on the source | |
| 12:27:17 | mriedem | lyarwood: is your plan for https://bugs.launchpad.net/nova/+bug/1832248 to get https://review.opendev.org/#/c/664418/ released and then bump the lower constraint dependency from nova to os-brick and then consider the nova bug fixed? | |
| 12:27:18 | openstack | Launchpad bug 1832248 in OpenStack Compute (nova) "tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume failing when using the Q35 machine type" [Undecided,New] | |
| 12:32:56 | alex_xu | mriedem: hope we answered all your question https://review.opendev.org/601596, looking for one more +2 :) | |
| 12:33:38 | alex_xu | johnthetubaguy: ^ hope you around, the vpmem spec is in good shape | |
| 12:34:19 | lyarwood | mriedem: no the nova bug is seperate, the os-brick change just works around the underlying QEMU issue Nova is hitting with q35 | |
| 12:36:31 | mriedem | lyarwood: oh ok | |
| 12:36:49 | mriedem | alex_xu: ack, i still need to read all of the replies... | |
| 12:37:20 | alex_xu | mriedem: hah, i see, a lot | |
| 12:39:45 | alex_xu | mriedem: also, there is the code for reference https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/virtual-persistent-memory, althought it is merge conflict, but it is still good to see what we probably are going to change in the code | |
| 12:42:59 | sean-k-mooney | mriedem: can you take a look at https://review.opendev.org/#/c/667264/ its a osc change for force down. you sent a mail to the list about droping computenode host/service id compat code and im wondering if that is related or not | |
| 12:43:45 | mriedem | i think my biggest hangups were on the (1) flavor extra spec definition which was a bit hard to parse from a user perspective in my opinion and (2) the questions about the new data model and versioned object which were very similar to a BDM but i realize we don't want to re-use BDMs for this | |
| 12:44:16 | mriedem | sean-k-mooney: different issue | |
| 12:44:31 | mriedem | sean-k-mooney: before 2.53 you had to call a force-down route, with 2.53 you just call the normal PUT route | |
| 12:44:39 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add mising tests for flavor extra_specs mv 2.61 https://review.opendev.org/667600 | |
| 12:44:48 | mriedem | https://developer.openstack.org/api-ref/compute/#update-forced-down for <2.53 | |
| 12:44:49 | sean-k-mooney | right i saw that | |
| 12:44:56 | mriedem | https://developer.openstack.org/api-ref/compute/#update-compute-service >=2.53 | |
| 12:45:05 | sean-k-mooney | what i was concerned about is the new form uses service id | |
| 12:45:06 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add missing tests for flavor extra_specs mv 2.61 https://review.opendev.org/667600 | |
| 12:45:06 | mriedem | with 2.53 the service_id in the API is a uuid | |
| 12:45:14 | sean-k-mooney | the old used host and binary name | |
| 12:45:29 | mriedem | service_id is the uuid of the service, it's fine | |
| 12:45:31 | sean-k-mooney | and i was not clear what you were proposeing droping in the mail | |
| 12:45:38 | mriedem | it's unrelated to the relationship between compute nodes and services | |
| 12:45:46 | sean-k-mooney | ok | |
| 12:45:56 | mriedem | see all of the notes/todos around ComputeNode.service_id in the code | |
| 12:46:00 | mriedem | and ComputeNode.host | |
| 12:46:22 | lyarwood | mriedem: https://review.opendev.org/#/c/457886/ - btw, would you mind taking a look at this if you have time this week. | |
| 12:46:48 | sean-k-mooney | mriedem: ok im reading them now thanks | |
| 12:47:13 | mriedem | lyarwood: sure | |
| 12:47:19 | lyarwood | thanks | |
| 13:00:40 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add missing tests for flavor extra_specs mv 2.61 https://review.opendev.org/667600 | |
| 13:08:52 | bauzas | mriedem: FWIW, I need to reload a shit ton of context from Kilo before replying to you but I saw your email | |
| 13:09:32 | bauzas | mriedem: because I wonder if we need a major version bump for the ComputeNode object | |
| 13:15:31 | mriedem | bauzas: i wondered about that as well but figured it wasn't required | |
| 13:15:53 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Implementing graceful shutdown. https://review.opendev.org/666245 | |
| 13:16:22 | mriedem | i think we've only ever bumped the major version on an object and that's when dansmith did Instance v2.0 | |
| 13:16:54 | mriedem | i don't remember the details of how complicated it was but i'm pretty sure i'd screw it up if i tried to do it myself | |
| 13:24:15 | bauzas | mriedem: yeah I need to remember why I was thinking about that by Kilo time | |
| 13:28:00 | mdbooth | stephenfin or sean-k-mooney: https://review.opendev.org/#/c/663382/4/nova/compute/manager.py Not my area of expertise, but would the prior call to _deallocate_network not mean that neutron would no longer return this stuff? | |
| 13:28:27 | mriedem | sean-k-mooney: i've replied on https://review.opendev.org/#/c/667264/2 with what i think they should do in the 2.53 case, | |
| 13:28:37 | mriedem | whether or not novaclient has all of the plumbing they need i haven't checked | |
| 13:29:28 | sean-k-mooney | mriedem: thanks osc is not what i normally review but since they asked me to take a look i said i would review | |
| 13:31:18 | mriedem | sean-k-mooney: mdbooth: also commented in https://review.opendev.org/#/c/663382/4 | |
| 13:31:50 | sean-k-mooney | maybe im looking. we could proably use try_dealocate_networks there too | |
| 13:31:59 | mdbooth | mriedem: Ooh, I'd forgotten that gem. | |
| 13:33:05 | mriedem | mdbooth: what? force_refresh? | |
| 13:33:07 | dansmith | mriedem: correct, and yes, it's complicated | |
| 13:33:08 | mriedem | you don't want to use that in this case | |
| 13:33:23 | mriedem | mdbooth: because force_refresh only goes back to stein and i'm guessing you want to backport this further than that | |
| 13:33:36 | mdbooth | mriedem: Ack. | |
| 13:43:52 | kashyap | Any others seeing stable/stein failures with the 'tempest-slow-p3' job? | |
| 13:44:01 | kashyap | http://logs.openstack.org/89/667389/1/check/tempest-slow-py3/2606bcc/testr_results.html.gz | |
| 13:44:38 | mriedem | kashyap: yes known issue | |
| 13:44:47 | mriedem | https://review.opendev.org/#/c/667216 | |
| 13:45:14 | kashyap | Ah, thanks. I didn't wanted to mindlessly do 'recheck' | |
| 13:46:55 | sean-k-mooney | mdbooth: deallocate_network delete the neutron port that were auto allcoated by nova so yes we proably should move that to the end of the function since it clears the network info caceh https://opendev.org/openstack/nova/src/branch/master/nova/network/neutronv2/api.py#L1603-L1604 | |
| 13:56:27 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add a rbd_connect_timeout configurable https://review.opendev.org/667421 | |
| 14:00:59 | mriedem | sean-k-mooney: i left some more comments/questions in that one and added some vmware and zvm driver devs | |
| 14:01:13 | sean-k-mooney | mriedem: it looks like https://review.opendev.org/#/c/660761/8 is trying to fix the same or a similar bug | |
| 14:03:53 | sean-k-mooney | mriedem: if we delete while building there is a scond race which causes us to not clean up the vif | |
| 14:04:22 | mriedem | that's amorin's fix yes | |
| 14:04:31 | mriedem | which is different from stephenfin's which is handling a failure while building | |
| 14:04:37 | sean-k-mooney | e.g. if the vm has spawned but we get teh delete before we update the instcance state in the db we raise an exception which is what cause us to not clean them up | |
| 14:04:42 | mriedem | and amorin was just in here the other day saying he had a similar issue there | |
| 14:05:17 | sean-k-mooney | mriedem: no stephens issue was a failure caused when you delete while building | |
| 14:05:52 | sean-k-mooney | spefically for the customer it was cause because one of the isntance in there heat stack failed to build and that cause all fo the instance to be deleted | |
| 14:07:35 | sean-k-mooney | mriedem: i think amorin bug is a duplicate of stephens but im not sure it would fix it in all cases as in sthepens edgecase we never call distroy | |
| 14:08:58 | sean-k-mooney | well maybe they are both bugs i did not fully review there bug in detail | |
| 14:12:17 | mriedem | as i said amorin said he still has an issue which stephenfin's patch might resolve | |