| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-13 | |||
| 01:40:22 | fungi | right now the only failure i see for nova is your functional-py35 job | |
| 01:41:21 | fungi | the problem is that when master branch jobs all switch to running on bionic by default, py35 jobs will still need to run on xenial because bionic doesn't provide python3.5 | |
| 01:41:58 | fungi | we solved this for any of the shared/abstract py35-specific jobs by setting them to use an ubuntu-xenial nodeset unconditionally | |
| 01:42:17 | fungi | but nova's functional-py35 job doesn't have any of those as a parent | |
| 01:42:42 | fungi | the options there are to change the parent on the job, or set an explicit nodeset in it to use ubuntu-xenial | |
| 01:43:18 | fungi | either fix should work and is only needed on master since the current stable branches already get their jobs set to use ubuntu-xenial | |
| 01:54:02 | melwitt | fungi: thanks | |
| 01:56:47 | fungi | one possible solution can be found in my edit to nova's .zuul.yaml in https://review.openstack.org/642580 | |
| 02:00:21 | melwitt | thanks | |
| 02:01:12 | fungi | nova gets this somewhat easier than other teams, since proposing changes to different branches of it is how i performed my preliminary testing | |
| 02:18:22 | openstackgerrit | Merged openstack/nova stable/queens: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/637599 | |
| 02:20:50 | openstackgerrit | melanie witt proposed openstack/nova master: Update nova-tox-functional-py35 parent job https://review.openstack.org/642940 | |
| 03:01:59 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Trivialfix for help description of images_type https://review.openstack.org/642942 | |
| 04:08:15 | openstackgerrit | Merged openstack/nova stable/queens: Don't emit warning when ironic properties are zero https://review.openstack.org/608611 | |
| 05:14:59 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic https://review.openstack.org/639017 | |
| 05:15:32 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic https://review.openstack.org/639017 | |
| 06:18:21 | openstackgerrit | Chris Friesen proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.openstack.org/639934 | |
| 09:07:23 | Kevin_Zheng | ping gmann | |
| 09:15:19 | dtantsur | efried: agreed! | |
| 09:51:59 | openstackgerrit | zhaixiaojun proposed openstack/nova master: Fix bug: non-admin user can not filter instance by instance-uuid https://review.openstack.org/642403 | |
| 09:55:33 | gibi | stephenfin: I'm wondering about the interference between the bandwidth feature and the numa aware vswitches feature | |
| 09:56:35 | gibi | stephenfin: is it a supported case that there are a single physnet with two ovs bridges one bridge connected to a nic on numa0 and the other bridge is connected to a nic on numa1? | |
| 09:57:14 | gibi | stephenfin: if it is supported how nova selects which bridge a port shall be plugged for this physnet? | |
| 10:16:10 | openstackgerrit | Merged openstack/nova stable/pike: Lock detach_volume https://review.openstack.org/637536 | |
| 10:43:31 | frickler | in case others have - like me - difficulties to decide between melwitt and efried for PTL, watch this ML thread before you vote: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003765.html | |
| 10:45:20 | openstackgerrit | Neil Jerram proposed openstack/nova stable/queens: Allow utime call to fail on qcow2 image base file https://review.openstack.org/642997 | |
| 10:47:03 | openstackgerrit | Neil Jerram proposed openstack/nova stable/rocky: Allow utime call to fail on qcow2 image base file https://review.openstack.org/642998 | |
| 10:49:08 | cdent | frickler: you might have more success if you ask some specific questions. Their nomination statements amount to "some statements explaining why they would be the better PTL" | |
| 10:49:47 | cdent | if you want them to differentiate themselves more, you need to ask them about a specific concern you have or present a scenario that you wonder how they would deal with | |
| 11:26:27 | mriedem | . | |
| 11:33:01 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Allow utime call to fail on qcow2 image base file https://review.openstack.org/643011 | |
| 11:50:50 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Move get_pci_mapping_for_migration to MigrationContext https://review.openstack.org/643023 | |
| 11:50:50 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Allow driver to properly unplug VIFs on destination on confirm resize https://review.openstack.org/643024 | |
| 12:08:40 | openstackgerrit | Merged openstack/nova master: pass endpoint interface to Ironic client https://review.openstack.org/640879 | |
| 12:11:10 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Add workaround to cleanup instance dir when using rbd https://review.openstack.org/628726 | |
| 12:24:44 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Allow utime call to fail on qcow2 image base file https://review.openstack.org/633493 | |
| 12:30:02 | mriedem | tonyb: what do you think about backporting this? https://review.openstack.org/#/c/641351/ | |
| 12:30:07 | mriedem | i asked in the ML as well, but no bites. | |
| 12:30:24 | mriedem | it's an ABI change but does fix a long-standing bug so admins don't shoot themselves | |
| 12:30:27 | mriedem | so i could go either way | |
| 12:32:06 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs Description https://review.openstack.org/643035 | |
| 12:35:26 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs string description for servers.py https://review.openstack.org/643035 | |
| 12:57:31 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs string description for servers.py https://review.openstack.org/643035 | |
| 13:01:14 | cdent | that's werid mriedem. is something perhaps sending allocations at the wrong microversion? | |
| 13:09:10 | mriedem | well it looks like it's sending a PUT /allocations request with no resources right? | |
| 13:09:59 | mriedem | and it also looks like it's only hitting in queens | |
| 13:10:02 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20save%20allocation%20for%5C%22%20AND%20message%3A%5C%22DestinationHypervisorTooOld%5C%22%20AND%20tags%3A%5C%22screen-n-super-cond.txt%5C%22&from=7d | |
| 13:10:47 | mriedem | i'm not sure why the hell it's logging DestinationHypervisorTooOld | |
| 13:11:12 | mriedem | Mar 13 01:08:15.816855 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: DEBUG nova.scheduler.client.report [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] Sending updated allocation [{'resource_provider': {'uuid': u'13509a14-0b25-4020-b1d6-2c56641a4d82'}, 'resources': {}}] for instance c2fd36f9-0b41-439c-9700-17aa0abe13c2 a | |
| 13:11:12 | mriedem | removing resources for 13509a14-0b25-4020-b1d6-2c56641a4d82. {{(pid=26101) remove_provider_from_instance_allocation /opt/stack/nova/nova/scheduler/client/report.py:1616}} | |
| 13:12:38 | mriedem | wtf, it thinks it's doing a same-host resize or something | |
| 13:19:18 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic https://review.openstack.org/639017 | |
| 13:26:23 | mriedem | oh heh, qemu is at different versions in this job | |
| 13:26:39 | mriedem | gmann: the tempest-slow job on stable/queens is running with different versions of ubuntu on the compute nodes | |
| 13:28:10 | mriedem | https://bugs.launchpad.net/tempest/+bug/1819910 | |
| 13:28:10 | openstack | Launchpad bug 1819910 in OpenStack Compute (nova) "test_server_connectivity_live_migration intermittently fails with NoValidHost due to PUT /allocations failure" [Undecided,Confirmed] | |
| 13:33:32 | mriedem | cdent: ok i figured out where that weird same-host allocation goofiness happens | |
| 13:34:26 | cdent | mriedem: sorry, only pay half attention, doing my taxes, which was a surprise gift for today | |
| 13:34:41 | mriedem | that is a fun gift | |
| 13:34:50 | mriedem | anyway, the allocations thing was a red herring | |
| 13:35:45 | mriedem | during live migration we call the scheduler for dest hosts and then do some checks on them in the conductor task, like the hypervisor version comparison between the source and dest, | |
| 13:35:56 | mriedem | if that fails, we call https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1527 to remove the allocations created on the dest provider by the scheduler, | |
| 13:36:13 | mriedem | if we get to https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1593 it assumes we're in a same-host resize situation, which is wrong | |
| 13:36:32 | mriedem | we end up sending {} for resource allocatoins and placement pukes on that https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1620 | |
| 13:36:54 | mriedem | we log a warning, and b/c we're doing this from an except block, oslo.log is also logging the original exception DestinationHypervisorTooOld - which makes it all very confusing | |
| 13:38:04 | frickler | mriedem: https://opendev.org/openstack-dev/devstack/src/branch/stable/queens/tools/fixup_stuff.sh#L86 | |
| 13:38:24 | frickler | seems ENABLE_VOLUME_MULTIATTACH is only set on controller and not on compute1 | |
| 13:39:11 | mriedem | frickler: gah that would do it | |
| 14:00:16 | gmann | mriedem: is it? | |
| 14:05:49 | mriedem | gmann: yeah https://bugs.launchpad.net/tempest/+bug/1819910 | |
| 14:05:50 | openstack | Launchpad bug 1819910 in tempest "test_server_connectivity_live_migration intermittently fails with NoValidHost due to DestinationHypervisorTooOld" [High,Triaged] | |
| 14:09:07 | mriedem | gmann: i'll try pushing a fix to tempest | |
| 14:11:23 | gmann | mriedem: thanks, is UCA thing will fix it. | |
| 14:14:10 | mriedem | gmann: ok i was working a patch but are you doing it now? | |
| 14:15:56 | gmann | mriedem: no, i am doing legacy job migration as of now. if you are working please push. | |
| 14:20:44 | mriedem | https://review.openstack.org/643052 | |
| 14:20:58 | mriedem | i'll push a nova stable/queens change that depends on ^ to make sure it works | |
| 14:28:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Fix live migration _remove_host_allocations warnings https://review.openstack.org/643055 | |
| 14:36:59 | openstackgerrit | Corey Bryant proposed openstack/nova master: add python 3.7 unit test job https://review.openstack.org/610694 | |
| 14:38:49 | brtknr | hello! | |
| 14:39:09 | brtknr | i have an instance stuck in BUILD which I do not seem to be able to delete... | |
| 14:39:23 | brtknr | whats the remedy | |
| 14:39:50 | brtknr | it appears when i do `openstack server list` but when i run `openstack server delete UUID`, i get: No server with a name or ID of '1a141848-c91a-450b-b4ad-6d2f81e4125e' exists. | |
| 14:40:37 | mriedem | brtknr: hmm, is there an entry for that instance_uuid in the nova_api.build_requests table? | |
| 14:41:00 | brtknr | in the database? | |
| 14:41:09 | brtknr | probably... | |
| 14:41:22 | brtknr | it exists as an entity somewhere... | |
| 14:41:26 | brtknr | but its not deletable | |
| 14:41:45 | mriedem | can you actually check the nova_api.build_requests table for a record with that instance_uuid? | |
| 14:41:52 | mriedem | because it should be temporary | |
| 14:43:51 | brtknr | mriedem: im checking, one sec | |
| 14:44:58 | brtknr | mysql> select * from build_requests; | |
| 14:44:59 | brtknr | Empty set (0.00 sec) | |
| 14:46:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: DNM: testing tempest-slow fix for subnode config https://review.openstack.org/643059 | |
| 14:47:37 | brtknr | mriedem: ive just deleted another server and its gone into the same state... still shows up under server list but cannot be deleted... | |
| 14:48:42 | mriedem | brtknr: so it's probably in a cell but the instance_mappings record in the nova_api database for it does not have cell_id set for some reason | |
| 14:48:51 | mriedem | check the nova_api.instance_mappings table for the instance_uuid in question, | |
| 14:49:01 | mriedem | the cell_id column should be set, but i'm guessing it's not in your case | |
| 14:49:44 | mriedem | that would suggest something is failing in your nova-conductor service during scheduling where the instance mapping is not updated to point at the cell in which the instance record actually exists | |
| 14:50:01 | mriedem | which is i'm guessing either cell0 or the 'nova' database (cell1) for you | |