| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-13 | |||
| 00:23:05 | melwitt | probably related to the legacy/deprecated nature of tls-proxy service | |
| 00:23:45 | melwitt | so tls-proxy isn't really related directly to our testing in nova-next, so since it's causing a problem, we can take it out. it wasn't helping us anymore anyway | |
| 00:23:55 | gmann | may be, we need to check if it is working in tempest-full. if so then we can wait to move nova-next on devstack based job | |
| 00:24:08 | gmann | sure. | |
| 00:24:48 | melwitt | gmann: nova-next is the only place where we try to test console proxy with tls. because the tls certs etc has to be setup | |
| 00:25:15 | openstackgerrit | Merged openstack/nova master: Allow utime call to fail on qcow2 image base file https://review.openstack.org/641315 | |
| 00:25:23 | openstackgerrit | Merged openstack/nova stable/rocky: Handle missing exception in instance creation code https://review.openstack.org/641401 | |
| 00:25:30 | openstackgerrit | Merged openstack/nova stable/rocky: Fix WeighedHost logging regression https://review.openstack.org/641355 | |
| 00:25:31 | melwitt | we just don't know the right way to do that nowadays. we used to piggyback on tls-proxy to do it | |
| 00:26:04 | gmann | ok | |
| 00:26:57 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic https://review.openstack.org/639017 | |
| 00:27:08 | gmann | melwitt: ^^ if it work then, i will mark nova OK in https://etherpad.openstack.org/p/legacy-job-bionic | |
| 00:27:45 | melwitt | gmann: OK, sounds good. thanks | |
| 00:28:12 | gmann | melwitt: thanks for help and updates on tls thing. | |
| 00:28:44 | melwitt | gmann: np, thanks for the heads up about the failure on bionic | |
| 00:42:02 | openstackgerrit | Merged openstack/nova stable/rocky: ironic: stop hammering ironic API in power sync loop https://review.openstack.org/640771 | |
| 00:42:08 | openstackgerrit | Merged openstack/nova stable/queens: Add description of custom resource classes https://review.openstack.org/619125 | |
| 01:08:29 | mnaser | efried: mind joining #openstack-tc -- just have some questions about powervm state | |
| 01:39:35 | melwitt | mriedem: I just noticed in #openstack-tc they're talking about jobs moving from xenial => bionic and project job failures. I don't understand most of what's in this email by fungi, but I think might know and be able to help: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003746.html | |
| 01:39:51 | melwitt | *I think you might know | |
| 01:40:04 | fungi | i am happy to explain too | |
| 01:40:22 | fungi | right now the only failure i see for nova is your functional-py35 job | |
| 01:41:21 | fungi | the problem is that when master branch jobs all switch to running on bionic by default, py35 jobs will still need to run on xenial because bionic doesn't provide python3.5 | |
| 01:41:58 | fungi | we solved this for any of the shared/abstract py35-specific jobs by setting them to use an ubuntu-xenial nodeset unconditionally | |
| 01:42:17 | fungi | but nova's functional-py35 job doesn't have any of those as a parent | |
| 01:42:42 | fungi | the options there are to change the parent on the job, or set an explicit nodeset in it to use ubuntu-xenial | |
| 01:43:18 | fungi | either fix should work and is only needed on master since the current stable branches already get their jobs set to use ubuntu-xenial | |
| 01:54:02 | melwitt | fungi: thanks | |
| 01:56:47 | fungi | one possible solution can be found in my edit to nova's .zuul.yaml in https://review.openstack.org/642580 | |
| 02:00:21 | melwitt | thanks | |
| 02:01:12 | fungi | nova gets this somewhat easier than other teams, since proposing changes to different branches of it is how i performed my preliminary testing | |
| 02:18:22 | openstackgerrit | Merged openstack/nova stable/queens: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/637599 | |
| 02:20:50 | openstackgerrit | melanie witt proposed openstack/nova master: Update nova-tox-functional-py35 parent job https://review.openstack.org/642940 | |
| 03:01:59 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Trivialfix for help description of images_type https://review.openstack.org/642942 | |
| 04:08:15 | openstackgerrit | Merged openstack/nova stable/queens: Don't emit warning when ironic properties are zero https://review.openstack.org/608611 | |
| 05:14:59 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic https://review.openstack.org/639017 | |
| 05:15:32 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic https://review.openstack.org/639017 | |
| 06:18:21 | openstackgerrit | Chris Friesen proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.openstack.org/639934 | |
| 09:07:23 | Kevin_Zheng | ping gmann | |
| 09:15:19 | dtantsur | efried: agreed! | |
| 09:51:59 | openstackgerrit | zhaixiaojun proposed openstack/nova master: Fix bug: non-admin user can not filter instance by instance-uuid https://review.openstack.org/642403 | |
| 09:55:33 | gibi | stephenfin: I'm wondering about the interference between the bandwidth feature and the numa aware vswitches feature | |
| 09:56:35 | gibi | stephenfin: is it a supported case that there are a single physnet with two ovs bridges one bridge connected to a nic on numa0 and the other bridge is connected to a nic on numa1? | |
| 09:57:14 | gibi | stephenfin: if it is supported how nova selects which bridge a port shall be plugged for this physnet? | |
| 10:16:10 | openstackgerrit | Merged openstack/nova stable/pike: Lock detach_volume https://review.openstack.org/637536 | |
| 10:43:31 | frickler | in case others have - like me - difficulties to decide between melwitt and efried for PTL, watch this ML thread before you vote: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003765.html | |
| 10:45:20 | openstackgerrit | Neil Jerram proposed openstack/nova stable/queens: Allow utime call to fail on qcow2 image base file https://review.openstack.org/642997 | |
| 10:47:03 | openstackgerrit | Neil Jerram proposed openstack/nova stable/rocky: Allow utime call to fail on qcow2 image base file https://review.openstack.org/642998 | |
| 10:49:08 | cdent | frickler: you might have more success if you ask some specific questions. Their nomination statements amount to "some statements explaining why they would be the better PTL" | |
| 10:49:47 | cdent | if you want them to differentiate themselves more, you need to ask them about a specific concern you have or present a scenario that you wonder how they would deal with | |
| 11:26:27 | mriedem | . | |
| 11:33:01 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Allow utime call to fail on qcow2 image base file https://review.openstack.org/643011 | |
| 11:50:50 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Allow driver to properly unplug VIFs on destination on confirm resize https://review.openstack.org/643024 | |
| 11:50:50 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Move get_pci_mapping_for_migration to MigrationContext https://review.openstack.org/643023 | |
| 12:08:40 | openstackgerrit | Merged openstack/nova master: pass endpoint interface to Ironic client https://review.openstack.org/640879 | |
| 12:11:10 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Add workaround to cleanup instance dir when using rbd https://review.openstack.org/628726 | |
| 12:24:44 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Allow utime call to fail on qcow2 image base file https://review.openstack.org/633493 | |
| 12:30:02 | mriedem | tonyb: what do you think about backporting this? https://review.openstack.org/#/c/641351/ | |
| 12:30:07 | mriedem | i asked in the ML as well, but no bites. | |
| 12:30:24 | mriedem | it's an ABI change but does fix a long-standing bug so admins don't shoot themselves | |
| 12:30:27 | mriedem | so i could go either way | |
| 12:32:06 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs Description https://review.openstack.org/643035 | |
| 12:35:26 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs string description for servers.py https://review.openstack.org/643035 | |
| 12:57:31 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs string description for servers.py https://review.openstack.org/643035 | |
| 13:01:14 | cdent | that's werid mriedem. is something perhaps sending allocations at the wrong microversion? | |
| 13:09:10 | mriedem | well it looks like it's sending a PUT /allocations request with no resources right? | |
| 13:09:59 | mriedem | and it also looks like it's only hitting in queens | |
| 13:10:02 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20save%20allocation%20for%5C%22%20AND%20message%3A%5C%22DestinationHypervisorTooOld%5C%22%20AND%20tags%3A%5C%22screen-n-super-cond.txt%5C%22&from=7d | |
| 13:10:47 | mriedem | i'm not sure why the hell it's logging DestinationHypervisorTooOld | |
| 13:11:12 | mriedem | removing resources for 13509a14-0b25-4020-b1d6-2c56641a4d82. {{(pid=26101) remove_provider_from_instance_allocation /opt/stack/nova/nova/scheduler/client/report.py:1616}} | |
| 13:11:12 | mriedem | Mar 13 01:08:15.816855 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: DEBUG nova.scheduler.client.report [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] Sending updated allocation [{'resource_provider': {'uuid': u'13509a14-0b25-4020-b1d6-2c56641a4d82'}, 'resources': {}}] for instance c2fd36f9-0b41-439c-9700-17aa0abe13c2 a | |
| 13:12:38 | mriedem | wtf, it thinks it's doing a same-host resize or something | |
| 13:19:18 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic https://review.openstack.org/639017 | |
| 13:26:23 | mriedem | oh heh, qemu is at different versions in this job | |
| 13:26:39 | mriedem | gmann: the tempest-slow job on stable/queens is running with different versions of ubuntu on the compute nodes | |
| 13:28:10 | openstack | Launchpad bug 1819910 in OpenStack Compute (nova) "test_server_connectivity_live_migration intermittently fails with NoValidHost due to PUT /allocations failure" [Undecided,Confirmed] | |
| 13:28:10 | mriedem | https://bugs.launchpad.net/tempest/+bug/1819910 | |
| 13:33:32 | mriedem | cdent: ok i figured out where that weird same-host allocation goofiness happens | |
| 13:34:26 | cdent | mriedem: sorry, only pay half attention, doing my taxes, which was a surprise gift for today | |
| 13:34:41 | mriedem | that is a fun gift | |
| 13:34:50 | mriedem | anyway, the allocations thing was a red herring | |
| 13:35:45 | mriedem | during live migration we call the scheduler for dest hosts and then do some checks on them in the conductor task, like the hypervisor version comparison between the source and dest, | |
| 13:35:56 | mriedem | if that fails, we call https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1527 to remove the allocations created on the dest provider by the scheduler, | |
| 13:36:13 | mriedem | if we get to https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1593 it assumes we're in a same-host resize situation, which is wrong | |
| 13:36:32 | mriedem | we end up sending {} for resource allocatoins and placement pukes on that https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1620 | |
| 13:36:54 | mriedem | we log a warning, and b/c we're doing this from an except block, oslo.log is also logging the original exception DestinationHypervisorTooOld - which makes it all very confusing | |
| 13:38:04 | frickler | mriedem: https://opendev.org/openstack-dev/devstack/src/branch/stable/queens/tools/fixup_stuff.sh#L86 | |
| 13:38:24 | frickler | seems ENABLE_VOLUME_MULTIATTACH is only set on controller and not on compute1 | |
| 13:39:11 | mriedem | frickler: gah that would do it | |
| 14:00:16 | gmann | mriedem: is it? | |
| 14:05:49 | mriedem | gmann: yeah https://bugs.launchpad.net/tempest/+bug/1819910 | |
| 14:05:50 | openstack | Launchpad bug 1819910 in tempest "test_server_connectivity_live_migration intermittently fails with NoValidHost due to DestinationHypervisorTooOld" [High,Triaged] | |
| 14:09:07 | mriedem | gmann: i'll try pushing a fix to tempest | |
| 14:11:23 | gmann | mriedem: thanks, is UCA thing will fix it. | |
| 14:14:10 | mriedem | gmann: ok i was working a patch but are you doing it now? | |
| 14:15:56 | gmann | mriedem: no, i am doing legacy job migration as of now. if you are working please push. | |
| 14:20:44 | mriedem | https://review.openstack.org/643052 | |
| 14:20:58 | mriedem | i'll push a nova stable/queens change that depends on ^ to make sure it works | |
| 14:28:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Fix live migration _remove_host_allocations warnings https://review.openstack.org/643055 | |
| 14:36:59 | openstackgerrit | Corey Bryant proposed openstack/nova master: add python 3.7 unit test job https://review.openstack.org/610694 | |