| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-18 | |||
| 22:42:49 | mriedem | melwitt: edleafe: ^ it's a start | |
| 22:42:57 | mriedem | gotta run | |
| 22:43:12 | edleafe | mriedem: ok, will post an update soon | |
| 22:43:30 | edleafe | and I renamed 'is_retry' to 'is_reschedule' | |
| 22:43:43 | edleafe | just seems more accurate | |
| 22:46:08 | melwitt | oh, cells v1 AND reschedules go through build_instances. guh | |
| 22:47:41 | edleafe | melwitt: fun, right? | |
| 22:48:08 | melwitt | totally | |
| 23:14:26 | rybridges | Hey guys | |
| 23:14:58 | rybridges | Any idea why running suspending an instance would put it into an error state? Here is the log from nova-compute https://pastebin.com/jTedyZVJ | |
| 23:15:48 | mgagne | can't wait for policy-in-code to land in all projects... =) | |
| 23:16:21 | mgagne | I guess that would be one major reason I would upgrade | |
| 23:16:40 | cfriesen | rybridges: is libvirtd running? | |
| 23:17:00 | rybridges | here is the output of server show https://pastebin.com/8eQiz3hA | |
| 23:17:03 | cfriesen | rybridges: check the libvirtd/qemu logs | |
| 23:17:04 | rybridges | yes libvirtd is running | |
| 23:17:38 | rybridges | this is happening intermittently | |
| 23:17:42 | rybridges | sometimes it happens | |
| 23:17:45 | rybridges | other times it is fine | |
| 23:18:56 | cfriesen | your nova logs indicate that the call to virDomainManagedSave() failed with a connection reset...this implies an issue with libvirtd, I think. | |
| 23:19:02 | cfriesen | I gotta run, suppertime | |
| 23:21:26 | rybridges | right | |
| 23:21:30 | rybridges | i figured that much | |
| 23:21:35 | rybridges | why would that call be failing? | |
| 23:21:42 | rybridges | i dont see anything in qemu logs | |
| 23:22:08 | rybridges | i am grepping for all sorts of stuff in /var/log/libvirt/qemu | |
| 23:22:17 | rybridges | the hypervisor is rhel7 btw | |
| 23:30:27 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add functional tests for resource class API https://review.openstack.org/524506 | |
| 23:31:15 | melwitt | rybridges: you'll want to look at the libvirt daemon logs, libvirtd.log. also look at /var/log/messages | |
| 23:31:37 | rybridges | by "libvirt daemon logs" do you mean the journalctl? | |
| 23:32:40 | melwitt | I'm most familiar with just looking at /var/log/libvirt/libvirtd.log but journalctl should show the same thing I think | |
| 23:35:02 | rybridges | i dont have anything in /var/log/libvirt/libvirtd.log | |
| 23:35:33 | melwitt | meaning, you don't have that file? your libvirtd might be configured to log somewhere else | |
| 23:35:40 | rybridges | the file is not there | |
| 23:35:44 | rybridges | yea i can check the conf | |
| 23:36:42 | clarkb | if it is going to journald you can do journalctl -u libvirtd (or whatever the unit name is) | |
| 23:37:00 | clarkb | the -u restricts it to printing logs only belonging to the listed units | |
| 23:37:31 | rybridges | right | |
| 23:37:35 | rybridges | i am using that.. | |
| 23:37:42 | rybridges | i will see if i find anything | |
| 23:41:22 | rybridges | o | |
| 23:41:26 | rybridges | ok* | |
| 23:41:58 | rybridges | libvirtd is showing: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error | |
| 23:51:33 | rybridges | am also seeing the following in some places: | |
| 23:51:35 | rybridges | virDBusCall:1558 : DBUS_METHOD_ERROR: 'org.freedesktop.machine1.Manager.TerminateMachine' on '/org/freedesktop/machine1' at 'org.freedesktop.machine1' error org.freedesktop.machine1.NoSuchMachine: No machine 'qemu-47-instance-000002e9' known | |
| 23:51:49 | rybridges | not sure if that is related | |
| 23:52:04 | rybridges | and i also see this: | |
| 23:52:06 | rybridges | error : qemuDomainObjEnterMonitorInternal:4003 : operation failed: domain is no longer running | |
| 23:54:22 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 23:54:23 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 23:54:30 | edleafe | mriedem: melwitt: ^^ moar updated | |
| #openstack-nova - 2017-12-19 | |||
| 00:00:40 | rybridges | not much seems pertinent in the libvirtd log | |
| 00:02:21 | rybridges | I dont think that this is related to libvirtd or qemu actually | |
| 00:02:44 | rybridges | we are running a juno deployment with identical libvirtd / qemu versions and config | |
| 00:02:50 | rybridges | and we do not see thsi problem | |
| 00:03:13 | rybridges | but when we run the same libvirtd/qemu setup with the ocata codebase, we see this issue | |
| 00:26:22 | rybridges | could be a problem with the libvirt-python version in ocata | |
| 00:26:35 | rybridges | the upper constraints is capped at 2.5.0 | |
| 00:26:46 | rybridges | but that is completely broken in rhel environments, cant even install it | |
| 00:26:52 | rybridges | so we tried 3.5.0 | |
| 00:26:56 | rybridges | that wasnt working | |
| 00:27:00 | rybridges | tried 3.10.0 | |
| 00:27:02 | rybridges | also not working | |
| 00:27:06 | rybridges | now trying 3.7.0 | |
| 00:27:09 | rybridges | and it seems to be working | |
| 00:27:19 | rybridges | i have suspended 40 instances without error | |
| 00:27:58 | rybridges | doh | |
| 00:28:01 | rybridges | take that back | |
| 00:28:06 | rybridges | tried 20 in parallel | |
| 00:28:09 | rybridges | still got a few errors | |
| 00:31:13 | clarkb | rybridges: libvirt-python is supposed to be compatible with any libvirt that is the same release as it or an older release. so libvirt-python 3.0 can tlak to libvirt 2.5 but libvirt-python 2.5 cn't talk to libvirt 3.0 | |
| 00:31:21 | clarkb | this is why rhel 7.4 broke the 2.5.0 cap | |
| 00:31:26 | clarkb | (they did a major upgrade of libvirt | |
| 00:32:22 | rybridges | right | |
| 00:32:24 | rybridges | yes | |
| 00:32:34 | rybridges | 2.5 breaks on later versions of libvirt | |
| 00:33:01 | rybridges | so we had to switch up | |
| 01:31:09 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/528881 | |
| 01:44:36 | mriedem | alex_xu: here is a question about something from long ago https://review.openstack.org/#/c/97727/ | |
| 01:44:38 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient master: Updated from global requirements https://review.openstack.org/528911 | |
| 01:44:59 | mriedem | alex_xu: why does populate_retry not check for MaxRetriesExceeded if max_attempts = 1? | |
| 01:45:23 | mriedem | i realize that means reschedules are disabled, but why wouldn't we compare num_attempts > max_attempts? | |
| 01:46:57 | mriedem | i guess that's what the code always did... | |
| 01:48:20 | mriedem | goes way back to https://review.openstack.org/#/c/9540/ | |
| 01:51:21 | mriedem | oh nvm, i know why | |
| 01:51:39 | mriedem | if max_attempts == 1, we never set the retry key in the filter properties passed to compute | |
| 01:52:08 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1855 | |
| 01:52:13 | mriedem | and then we don't reschedule | |
| 01:53:17 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/528881 | |
| 01:53:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't try to delete build requests on MaxRetriesExceeded https://review.openstack.org/528835 | |
| 01:53:45 | mriedem | melwitt: % | |
| 01:53:46 | mriedem | ^ | |
| 01:54:36 | mriedem | gah, that also goes back to newton | |
| 02:16:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't try to delete build request during a reschedule https://review.openstack.org/528835 | |
| 02:17:32 | rybridges | so I think i found the root problem with suspend | |
| 02:17:44 | rybridges | when i run suspend like this: openstack server suspend <uuid1> <uuid2> <uuid3> <uuid4> <uuid5>..... <uuid40> | |
| 02:17:51 | rybridges | almost all of the instances go to error state | |
| 02:17:53 | rybridges | but | |
| 02:18:30 | rybridges | when I run suspend in a simple for loop | |
| 02:18:33 | rybridges | like this: | |