| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-13 | |||
| 12:30:02 | mriedem | tonyb: what do you think about backporting this? https://review.openstack.org/#/c/641351/ | |
| 12:30:07 | mriedem | i asked in the ML as well, but no bites. | |
| 12:30:24 | mriedem | it's an ABI change but does fix a long-standing bug so admins don't shoot themselves | |
| 12:30:27 | mriedem | so i could go either way | |
| 12:32:06 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs Description https://review.openstack.org/643035 | |
| 12:35:26 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs string description for servers.py https://review.openstack.org/643035 | |
| 12:57:31 | openstackgerrit | chenker proposed openstack/python-novaclient master: Optimize limit option docs string description for servers.py https://review.openstack.org/643035 | |
| 13:01:14 | cdent | that's werid mriedem. is something perhaps sending allocations at the wrong microversion? | |
| 13:09:10 | mriedem | well it looks like it's sending a PUT /allocations request with no resources right? | |
| 13:09:59 | mriedem | and it also looks like it's only hitting in queens | |
| 13:10:02 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20save%20allocation%20for%5C%22%20AND%20message%3A%5C%22DestinationHypervisorTooOld%5C%22%20AND%20tags%3A%5C%22screen-n-super-cond.txt%5C%22&from=7d | |
| 13:10:47 | mriedem | i'm not sure why the hell it's logging DestinationHypervisorTooOld | |
| 13:11:12 | mriedem | removing resources for 13509a14-0b25-4020-b1d6-2c56641a4d82. {{(pid=26101) remove_provider_from_instance_allocation /opt/stack/nova/nova/scheduler/client/report.py:1616}} | |
| 13:11:12 | mriedem | Mar 13 01:08:15.816855 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: DEBUG nova.scheduler.client.report [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] Sending updated allocation [{'resource_provider': {'uuid': u'13509a14-0b25-4020-b1d6-2c56641a4d82'}, 'resources': {}}] for instance c2fd36f9-0b41-439c-9700-17aa0abe13c2 a | |
| 13:12:38 | mriedem | wtf, it thinks it's doing a same-host resize or something | |
| 13:19:18 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic https://review.openstack.org/639017 | |
| 13:26:23 | mriedem | oh heh, qemu is at different versions in this job | |
| 13:26:39 | mriedem | gmann: the tempest-slow job on stable/queens is running with different versions of ubuntu on the compute nodes | |
| 13:28:10 | openstack | Launchpad bug 1819910 in OpenStack Compute (nova) "test_server_connectivity_live_migration intermittently fails with NoValidHost due to PUT /allocations failure" [Undecided,Confirmed] | |
| 13:28:10 | mriedem | https://bugs.launchpad.net/tempest/+bug/1819910 | |
| 13:33:32 | mriedem | cdent: ok i figured out where that weird same-host allocation goofiness happens | |
| 13:34:26 | cdent | mriedem: sorry, only pay half attention, doing my taxes, which was a surprise gift for today | |
| 13:34:41 | mriedem | that is a fun gift | |
| 13:34:50 | mriedem | anyway, the allocations thing was a red herring | |
| 13:35:45 | mriedem | during live migration we call the scheduler for dest hosts and then do some checks on them in the conductor task, like the hypervisor version comparison between the source and dest, | |
| 13:35:56 | mriedem | if that fails, we call https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1527 to remove the allocations created on the dest provider by the scheduler, | |
| 13:36:13 | mriedem | if we get to https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1593 it assumes we're in a same-host resize situation, which is wrong | |
| 13:36:32 | mriedem | we end up sending {} for resource allocatoins and placement pukes on that https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1620 | |
| 13:36:54 | mriedem | we log a warning, and b/c we're doing this from an except block, oslo.log is also logging the original exception DestinationHypervisorTooOld - which makes it all very confusing | |
| 13:38:04 | frickler | mriedem: https://opendev.org/openstack-dev/devstack/src/branch/stable/queens/tools/fixup_stuff.sh#L86 | |
| 13:38:24 | frickler | seems ENABLE_VOLUME_MULTIATTACH is only set on controller and not on compute1 | |
| 13:39:11 | mriedem | frickler: gah that would do it | |
| 14:00:16 | gmann | mriedem: is it? | |
| 14:05:49 | mriedem | gmann: yeah https://bugs.launchpad.net/tempest/+bug/1819910 | |
| 14:05:50 | openstack | Launchpad bug 1819910 in tempest "test_server_connectivity_live_migration intermittently fails with NoValidHost due to DestinationHypervisorTooOld" [High,Triaged] | |
| 14:09:07 | mriedem | gmann: i'll try pushing a fix to tempest | |
| 14:11:23 | gmann | mriedem: thanks, is UCA thing will fix it. | |
| 14:14:10 | mriedem | gmann: ok i was working a patch but are you doing it now? | |
| 14:15:56 | gmann | mriedem: no, i am doing legacy job migration as of now. if you are working please push. | |
| 14:20:44 | mriedem | https://review.openstack.org/643052 | |
| 14:20:58 | mriedem | i'll push a nova stable/queens change that depends on ^ to make sure it works | |
| 14:28:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Fix live migration _remove_host_allocations warnings https://review.openstack.org/643055 | |
| 14:36:59 | openstackgerrit | Corey Bryant proposed openstack/nova master: add python 3.7 unit test job https://review.openstack.org/610694 | |
| 14:38:49 | brtknr | hello! | |
| 14:39:09 | brtknr | i have an instance stuck in BUILD which I do not seem to be able to delete... | |
| 14:39:23 | brtknr | whats the remedy | |
| 14:39:50 | brtknr | it appears when i do `openstack server list` but when i run `openstack server delete UUID`, i get: No server with a name or ID of '1a141848-c91a-450b-b4ad-6d2f81e4125e' exists. | |
| 14:40:37 | mriedem | brtknr: hmm, is there an entry for that instance_uuid in the nova_api.build_requests table? | |
| 14:41:00 | brtknr | in the database? | |
| 14:41:09 | brtknr | probably... | |
| 14:41:22 | brtknr | it exists as an entity somewhere... | |
| 14:41:26 | brtknr | but its not deletable | |
| 14:41:45 | mriedem | can you actually check the nova_api.build_requests table for a record with that instance_uuid? | |
| 14:41:52 | mriedem | because it should be temporary | |
| 14:43:51 | brtknr | mriedem: im checking, one sec | |
| 14:44:58 | brtknr | mysql> select * from build_requests; | |
| 14:44:59 | brtknr | Empty set (0.00 sec) | |
| 14:46:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: DNM: testing tempest-slow fix for subnode config https://review.openstack.org/643059 | |
| 14:47:37 | brtknr | mriedem: ive just deleted another server and its gone into the same state... still shows up under server list but cannot be deleted... | |
| 14:48:42 | mriedem | brtknr: so it's probably in a cell but the instance_mappings record in the nova_api database for it does not have cell_id set for some reason | |
| 14:48:51 | mriedem | check the nova_api.instance_mappings table for the instance_uuid in question, | |
| 14:49:01 | mriedem | the cell_id column should be set, but i'm guessing it's not in your case | |
| 14:49:44 | mriedem | that would suggest something is failing in your nova-conductor service during scheduling where the instance mapping is not updated to point at the cell in which the instance record actually exists | |
| 14:50:01 | mriedem | which is i'm guessing either cell0 or the 'nova' database (cell1) for you | |
| 14:57:20 | openstackgerrit | Merged openstack/nova stable/pike: [pike-only] Fix resize_instance rpcapi call https://review.openstack.org/603439 | |
| 14:57:27 | openstackgerrit | Merged openstack/nova stable/pike: tox: Don't write byte code (maybe) https://review.openstack.org/636919 | |
| 14:57:36 | openstackgerrit | Merged openstack/nova stable/pike: PCI: do not force remove allocated devices https://review.openstack.org/635074 | |
| 14:57:46 | openstackgerrit | Merged openstack/nova stable/pike: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623552 | |
| 14:58:47 | dansmith | wow, pike | |
| 15:15:10 | mriedem | dansmith: if you like those, mayhap you want to hit these other rocky +2'ed changes https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/rocky+label:Code-Review=2 | |
| 15:16:25 | dansmith | no good deed | |
| 15:23:07 | openstackgerrit | Merged openstack/nova stable/rocky: Handle templated cell mappings in nova-status https://review.openstack.org/642611 | |
| 15:26:14 | dansmith | mriedem: I got stopped yesterday with that bigger ironic change at the top there | |
| 15:27:06 | mriedem | just plow right through those concerns | |
| 15:29:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance to make place for port healing https://review.openstack.org/637953 | |
| 15:29:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.openstack.org/637955 | |
| 15:29:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance (2) https://review.openstack.org/637954 | |
| 15:29:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: cache neutron ports in heal allocation https://review.openstack.org/638207 | |
| 15:43:00 | openstackgerrit | Merged openstack/nova master: Remove duplicate cleanup in functional tests https://review.openstack.org/636996 | |
| 15:47:38 | mriedem | gibi: i went over the docs for bw providers https://review.openstack.org/#/c/642064/ some comments inline | |
| 15:47:46 | mriedem | let me know if you want to discuss my comments | |
| 15:47:52 | gibi | mriedem: thanks. looking | |
| 15:49:41 | gmann | melwitt: mriedem nova-next is failing with same reason - http://logs.openstack.org/17/639017/8/check/nova-next/21327c4/logs/screen-n-cpu.txt.gz#_Mar_13_14_31_14_886123 | |
| 15:51:26 | mriedem | gmann: i don't think that's the same, | |
| 15:51:29 | mriedem | and i don't think nova-next is multinode | |
| 15:52:06 | mriedem | http://logs.openstack.org/17/639017/8/check/nova-next/21327c4/logs/libvirt/qemu/instance-00000001.txt.gz | |
| 15:52:11 | gmann | mriedem: not same as tempest-slow. i mean cert issue which was occurring early starting from when migrating it to bionic | |
| 15:52:13 | mriedem | -vnc 127.0.0.1:0,tls,x509verify=/etc/pki/libvirt-vnc: Failed to start VNC server: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file. | |
| 15:52:21 | gmann | yeah this one. | |
| 15:52:42 | gmann | melwitt: was trying it with 1024 bit. | |
| 15:53:02 | mriedem | ok i don't know about the vnc / cert stuff | |
| 15:53:21 | gmann | mriedem: sorry for mixing it. it is for https://review.openstack.org/#/c/639017/ | |
| 15:55:37 | cfriesen | I think this should be a fairly uncontroversial bugfix (has +2 from efried): https://review.openstack.org/#/c/641480/ | |
| 15:58:26 | mriedem | dansmith: you'll never believe it by the $11 lady won | |
| 15:58:29 | mriedem | *but the | |
| 15:58:36 | mriedem | i've been carole'ed | |
| 15:58:47 | dansmith | mriedem: now you have to explain to maya why she's not an idiot after all? | |
| 15:58:56 | mriedem | i think she's sleeping | |
| 15:59:12 | dansmith | heh | |
| 16:00:28 | mriedem | too much colonial penn commercials | |