| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-18 | |||
| 22:26:49 | edleafe | ok, I'll move the check there | |
| 22:27:49 | mriedem | note that in that case, we shouldn' try to destroy the build request in the except block | |
| 22:28:14 | edleafe | why not? | |
| 22:28:20 | mriedem | because if we're in a cell conductor we don't have access to the api db | |
| 22:28:32 | mriedem | and rather than BuildRequestNotFound, we'll get CantStartEngineError | |
| 22:28:41 | mriedem | ala https://bugs.launchpad.net/nova/+bug/1733933 | |
| 22:28:41 | openstack | Launchpad bug 1733933 in OpenStack Compute (nova) "nova-conductor is masking error when rescheduling" [Low,Triaged] | |
| 22:29:00 | edleafe | ah. I don't know how you keep all that straight | |
| 22:29:27 | mriedem | well, that's also what failed here http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cond-cell1.txt.gz#_Dec_12_22_23_48_751837 | |
| 22:29:41 | mriedem | er this i mean http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cond-cell1.txt.gz#_Dec_12_22_23_48_940236 | |
| 22:29:52 | mriedem | File "/opt/stack/new/nova/nova/conductor/manager.py", line 478, in _destroy_build_request | |
| 22:30:26 | mriedem | i'll leave a note in the code so we don't forget | |
| 22:34:07 | mriedem | edleafe: ok commented. after thinking through that, it might be best to leave L578 | |
| 22:34:17 | mriedem | because host_lists is going to be per-instance right? | |
| 22:34:31 | mriedem | although, if we're rescheduling, it will be a single entry yeah? | |
| 22:35:23 | edleafe | yeah, always single on reschedule | |
| 22:36:34 | edleafe | also, the populate_retry method pops the exc_reason from the filter_properties, so it isn't available later in build_instances | |
| 22:36:56 | edleafe | why not check above as you mentioned before? | |
| 22:37:04 | mriedem | we could - i left yet another comment | |
| 22:37:15 | mriedem | i think the build request cleanup we have in here is likely latent broken | |
| 22:37:19 | mriedem | if we're rescheduling | |
| 22:37:43 | mriedem | if we get here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L559 because populate_retry raised MaxRetriesExceeded, | |
| 22:37:50 | mriedem | we shouldn't try to delete the build request | |
| 22:38:01 | mriedem | melwitt: you might want to check my thinking here | |
| 22:41:00 | melwitt | yeah ... the way that code is written, it's assuming we have an instance record. and I thought we do indeed have an instance in ERROR state if max retries were reached | |
| 22:41:53 | melwitt | I can't remember | |
| 22:42:46 | melwitt | also, that's cells v1 only code where I think the instance record is created right away in the compute/api. so a retries exceeded would have an instance record | |
| 22:42:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Don't try to delete build requests on reschedules exceeded https://review.openstack.org/528835 | |
| 22:42:49 | mriedem | melwitt: edleafe: ^ it's a start | |
| 22:42:57 | mriedem | gotta run | |
| 22:43:12 | edleafe | mriedem: ok, will post an update soon | |
| 22:43:30 | edleafe | and I renamed 'is_retry' to 'is_reschedule' | |
| 22:43:43 | edleafe | just seems more accurate | |
| 22:46:08 | melwitt | oh, cells v1 AND reschedules go through build_instances. guh | |
| 22:47:41 | edleafe | melwitt: fun, right? | |
| 22:48:08 | melwitt | totally | |
| 23:14:26 | rybridges | Hey guys | |
| 23:14:58 | rybridges | Any idea why running suspending an instance would put it into an error state? Here is the log from nova-compute https://pastebin.com/jTedyZVJ | |
| 23:15:48 | mgagne | can't wait for policy-in-code to land in all projects... =) | |
| 23:16:21 | mgagne | I guess that would be one major reason I would upgrade | |
| 23:16:40 | cfriesen | rybridges: is libvirtd running? | |
| 23:17:00 | rybridges | here is the output of server show https://pastebin.com/8eQiz3hA | |
| 23:17:03 | cfriesen | rybridges: check the libvirtd/qemu logs | |
| 23:17:04 | rybridges | yes libvirtd is running | |
| 23:17:38 | rybridges | this is happening intermittently | |
| 23:17:42 | rybridges | sometimes it happens | |
| 23:17:45 | rybridges | other times it is fine | |
| 23:18:56 | cfriesen | your nova logs indicate that the call to virDomainManagedSave() failed with a connection reset...this implies an issue with libvirtd, I think. | |
| 23:19:02 | cfriesen | I gotta run, suppertime | |
| 23:21:26 | rybridges | right | |
| 23:21:30 | rybridges | i figured that much | |
| 23:21:35 | rybridges | why would that call be failing? | |
| 23:21:42 | rybridges | i dont see anything in qemu logs | |
| 23:22:08 | rybridges | i am grepping for all sorts of stuff in /var/log/libvirt/qemu | |
| 23:22:17 | rybridges | the hypervisor is rhel7 btw | |
| 23:30:27 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add functional tests for resource class API https://review.openstack.org/524506 | |
| 23:31:15 | melwitt | rybridges: you'll want to look at the libvirt daemon logs, libvirtd.log. also look at /var/log/messages | |
| 23:31:37 | rybridges | by "libvirt daemon logs" do you mean the journalctl? | |
| 23:32:40 | melwitt | I'm most familiar with just looking at /var/log/libvirt/libvirtd.log but journalctl should show the same thing I think | |
| 23:35:02 | rybridges | i dont have anything in /var/log/libvirt/libvirtd.log | |
| 23:35:33 | melwitt | meaning, you don't have that file? your libvirtd might be configured to log somewhere else | |
| 23:35:40 | rybridges | the file is not there | |
| 23:35:44 | rybridges | yea i can check the conf | |
| 23:36:42 | clarkb | if it is going to journald you can do journalctl -u libvirtd (or whatever the unit name is) | |
| 23:37:00 | clarkb | the -u restricts it to printing logs only belonging to the listed units | |
| 23:37:31 | rybridges | right | |
| 23:37:35 | rybridges | i am using that.. | |
| 23:37:42 | rybridges | i will see if i find anything | |
| 23:41:22 | rybridges | o | |
| 23:41:26 | rybridges | ok* | |
| 23:41:58 | rybridges | libvirtd is showing: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error | |
| 23:51:33 | rybridges | am also seeing the following in some places: | |
| 23:51:35 | rybridges | virDBusCall:1558 : DBUS_METHOD_ERROR: 'org.freedesktop.machine1.Manager.TerminateMachine' on '/org/freedesktop/machine1' at 'org.freedesktop.machine1' error org.freedesktop.machine1.NoSuchMachine: No machine 'qemu-47-instance-000002e9' known | |
| 23:51:49 | rybridges | not sure if that is related | |
| 23:52:04 | rybridges | and i also see this: | |
| 23:52:06 | rybridges | error : qemuDomainObjEnterMonitorInternal:4003 : operation failed: domain is no longer running | |
| 23:54:22 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 23:54:23 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 23:54:30 | edleafe | mriedem: melwitt: ^^ moar updated | |
| #openstack-nova - 2017-12-19 | |||
| 00:00:40 | rybridges | not much seems pertinent in the libvirtd log | |
| 00:02:21 | rybridges | I dont think that this is related to libvirtd or qemu actually | |
| 00:02:44 | rybridges | we are running a juno deployment with identical libvirtd / qemu versions and config | |
| 00:02:50 | rybridges | and we do not see thsi problem | |
| 00:03:13 | rybridges | but when we run the same libvirtd/qemu setup with the ocata codebase, we see this issue | |
| 00:26:22 | rybridges | could be a problem with the libvirt-python version in ocata | |
| 00:26:35 | rybridges | the upper constraints is capped at 2.5.0 | |
| 00:26:46 | rybridges | but that is completely broken in rhel environments, cant even install it | |
| 00:26:52 | rybridges | so we tried 3.5.0 | |
| 00:26:56 | rybridges | that wasnt working | |
| 00:27:00 | rybridges | tried 3.10.0 | |
| 00:27:02 | rybridges | also not working | |
| 00:27:06 | rybridges | now trying 3.7.0 | |
| 00:27:09 | rybridges | and it seems to be working | |
| 00:27:19 | rybridges | i have suspended 40 instances without error | |
| 00:27:58 | rybridges | doh | |
| 00:28:01 | rybridges | take that back | |
| 00:28:06 | rybridges | tried 20 in parallel | |
| 00:28:09 | rybridges | still got a few errors | |
| 00:31:13 | clarkb | rybridges: libvirt-python is supposed to be compatible with any libvirt that is the same release as it or an older release. so libvirt-python 3.0 can tlak to libvirt 2.5 but libvirt-python 2.5 cn't talk to libvirt 3.0 | |
| 00:31:21 | clarkb | this is why rhel 7.4 broke the 2.5.0 cap | |