| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-03 | |||
| 23:26:38 | mriedem | stvnoyes: running the test again based on these changes to the libvirt patch ^ | |
| #openstack-nova - 2018-01-04 | |||
| 00:23:10 | mriedem | nailed it http://logs.openstack.org/67/529867/1/check/tempest-full/23d2919/controller/logs/screen-n-sch.txt.gz#_Dec_22_18_14_13_693048 | |
| 00:23:34 | mriedem | jaypipes: could you have guessed the stats / num_instances / host_state.instances stuff could have all gotten screwy? | |
| 00:26:58 | jaypipes | mriedem: not surprised. | |
| 00:31:00 | mriedem | heh "Reported number of instances (0) does not match the tracked number of instances (3)." | |
| 00:31:03 | mriedem | not even close | |
| 00:38:44 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: doc: Add configuration index page https://review.openstack.org/531042 | |
| 00:54:21 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.59 - List/Show all server migration types https://review.openstack.org/430839 | |
| 01:02:45 | mriedem | will need someone more familiar with ironic to triage this https://bugs.launchpad.net/nova/+bug/1730834 | |
| 01:02:46 | openstack | Launchpad bug 1730834 in OpenStack Compute (nova) "Ironic compute node doesn't take over nodes with instance when the owner compute node is down" [Undecided,New] | |
| 01:02:59 | mriedem | i don't know what "take over" means here | |
| 01:03:35 | jroll | mriedem: that's expected behavior | |
| 01:03:57 | jroll | 'take over' meaning have another compute service manage the instance, because the hash ring thing | |
| 01:04:37 | jroll | we should make that better so it isn't expected behavior, but low priority I guess | |
| 01:04:43 | mriedem | oh | |
| 01:04:44 | mriedem | well then | |
| 01:07:00 | jroll | oh, I can't set importance ¯\_(ツ)_/¯ | |
| 01:15:56 | mriedem | join the bug team | |
| 01:15:58 | mriedem | should be able to then | |
| 01:17:25 | jroll | idk, you might make me do stuff | |
| 01:26:51 | bjhuangr | mriedem, hi, do you have a chance to review https://review.openstack.org/#/c/523387/ ? Thanks in advance . | |
| 01:33:37 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: [libvirt] Add _get_XXXpin_cpuset() https://review.openstack.org/527631 | |
| 01:33:38 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add NumaTopology support for libvirt/qemu driver https://review.openstack.org/530451 | |
| 01:33:38 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: disable cpu pinning with libvirt/qemu driver https://review.openstack.org/531049 | |
| 02:20:58 | mriedem | kashyap: could use some help with this if you get a chance https://review.openstack.org/#/c/267587/75/nova/virt/libvirt/guest.py - trying to get multiattach working which used to be ok before qemu 2.10 but now we hit issues with a write lock when attaching the volume to the 2nd guest, and i thought we could pass the force flag to the attach device call to libvirt but i got this error: | |
| 02:21:04 | mriedem | libvirtError: unsupported flags (0x4) in function qemuDomainAttachDeviceLiveAndConfig | |
| 02:25:11 | Kevin_Zheng | seems Nova is broken after add uuid to BDM | |
| 02:25:16 | Kevin_Zheng | due to https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L497 | |
| 02:25:31 | openstackgerrit | Lance Bragstad proposed openstack/nova master: Simplify logic in get_enforcer https://review.openstack.org/531008 | |
| 02:25:48 | Kevin_Zheng | the field could not be added to cell1 db | |
| 02:27:05 | mriedem | Kevin_Zheng: that nova-manage code is really old | |
| 02:27:17 | mriedem | if the bdm uuid change broke that, we should have seen it in CI | |
| 02:27:19 | mriedem | since devstack runs this | |
| 02:27:38 | mriedem | the cell1 sync happens here https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L513 | |
| 02:27:45 | mriedem | after running the schema migrations for cell0 | |
| 02:28:46 | Kevin_Zheng | Hmm, but my db didn't got updated | |
| 02:28:53 | mriedem | is your nova.conf correct? | |
| 02:30:34 | Kevin_Zheng | I will check, but I have been using this for sometime, it should be correct. | |
| 02:32:11 | Kevin_Zheng | Hmm... in config file for api service, the database connection should be cell0 db, correct? | |
| 02:35:52 | mriedem | that's what we have in http://logs.openstack.org/58/526258/3/check/tempest-full/d133d1f/controller/logs/etc/nova/ | |
| 02:36:04 | mriedem | but devstack will sync using the cell1 conf too | |
| 02:36:11 | mriedem | which has the cell1 database in the [database] section | |
| 02:36:48 | mriedem | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L707 | |
| 02:37:08 | mriedem | this runs it for cell0 https://github.com/openstack-dev/devstack/blob/master/lib/nova#L711 | |
| 02:37:29 | Kevin_Zheng | ah, I see | |
| 02:37:48 | mriedem | this was never implemented to hit all cells https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L479 | |
| 02:37:53 | mriedem | i had a patch for it but it must be abandoned | |
| 02:38:22 | mriedem | https://review.openstack.org/#/c/420973/ | |
| 02:39:26 | Kevin_Zheng | yeah, | |
| 02:39:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix up formatting for deprecate-api-extensions-policies release note https://review.openstack.org/531061 | |
| 02:39:40 | Kevin_Zheng | Thanks, problem solved | |
| 02:39:53 | Kevin_Zheng | did we mentioned this in any docs? | |
| 02:41:10 | mriedem | it should be in the install guide | |
| 02:42:22 | mriedem | oh, well, https://docs.openstack.org/nova/latest/install/controller-install-ubuntu.html#install-and-configure-components | |
| 02:42:25 | melwitt | mriedem: re: that multiattach thing, did you see this bug? https://bugzilla.redhat.com/show_bug.cgi?id=1378242 based on that it looks like there needs to be share-rw=on property set under |
|
| 02:42:26 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:42:40 | mriedem | Kevin_Zheng: that install guide works because it's configuring nova.conf to set the [database] to the cell1 db | |
| 02:42:56 | mriedem | Kevin_Zheng: that install guide was written before the superconductor mode stuff that dansmith did in devstack | |
| 02:43:04 | mriedem | where the controllers are pointed at cell0 | |
| 02:43:22 | mriedem | Kevin_Zheng: we also have https://docs.openstack.org/nova/latest/user/cells.html#setup-of-cells-v2 | |
| 02:44:12 | mriedem | it's not very clear, but there is a note in there too | |
| 02:44:13 | mriedem | "At this point, the API database can now find the cell database, and further commands will attempt to look inside. If this is a completely fresh database (such as if you’re adding a cell, or if this is a new deployment), then you will need to run nova-manage db sync on it to initialize the schema." | |
| 02:44:31 | mriedem | Kevin_Zheng: might be a good FAQs entry https://docs.openstack.org/nova/latest/user/cells.html#faqs | |
| 02:44:43 | mriedem | melwitt: nope never seen that | |
| 02:44:59 | Kevin_Zheng | yeah, | |
| 02:45:07 | Kevin_Zheng | I will add it later | |
| 02:45:40 | mriedem | melwitt: not mentioned in the domain xml docs at all https://libvirt.org/formatdomain.html | |
| 02:45:41 | mriedem | :( | |
| 02:47:27 | mriedem | and by the looks of when this was 'fixed' i'm guessing we'd need super modern versions of qemu to use this | |
| 02:47:45 | melwitt | yeah, I was just thinking the same | |
| 02:48:15 | mriedem | well, so much for multiattach in queens | |
| 02:48:21 | melwitt | this all looks fairly recent, which would explain the absence in the docs | |
| 02:48:22 | mriedem | wah wah | |
| 02:48:34 | mriedem | yeah | |
| 02:48:35 | mriedem | commit 860a3c4bea1d24773d8a495f213d5de3ac48a462 Author: Peter Krempa <pkrempa@redhat.com> Date: Wed Nov 15 15:02:58 2017 +0100 | |
| 02:50:22 | mriedem | we could have conditional logic around the version of qemu being used, but that sucks, | |
| 02:50:36 | mriedem | and i don't know if like, qemu < 2.10 works, and then is broken until qemu >= x | |
| 02:50:47 | mriedem | so you'd have a middle ground where things just don't work | |
| 02:53:08 | melwitt | what do you mean, like if this was a regression there could be a window in the middle where things don't work? | |
| 02:53:13 | mriedem | yeah | |
| 02:53:27 | mriedem | we didn't have this problem around ~newton when i wrote the original tempest test for multiattach | |
| 02:53:40 | mriedem | you could do the 2 attachments fine, it's just that nova didn't orchestrate the detach properly | |
| 02:53:45 | melwitt | oh :( | |
| 02:53:48 | mriedem | now we're in a case with newer qemu where the 2nd attach fails | |
| 02:54:07 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: log test: use fixtures.StandardLogging in setUp https://review.openstack.org/531065 | |
| 02:54:29 | mriedem | so apparently fixed in libvirt-3.9.0-3.el7 | |
| 02:54:53 | mriedem | we are testing against 3.6.0 | |
| 02:55:34 | mriedem | https://bugzilla.redhat.com/show_bug.cgi?id=1378242#c13 | |
| 02:55:35 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:55:38 | melwitt | and this is the qemu bug that the libvirt bug was cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1378241 | |
| 02:55:38 | openstack | bugzilla.redhat.com bug 1378241 in qemu-kvm-rhev "QEMU image file locking" [Unspecified,Verified] - Assigned to famz | |
| 02:55:38 | mriedem | same thing we're hitting | |
| 02:56:21 | melwitt | that says qemu-kvm-rhev though. | |
| 02:57:11 | mriedem | yeah so that caused the bug we had with 2.10 where we were hitting the same issue with qemu-img info | |
| 02:57:17 | mriedem | and had to start using the force flag for that | |
| 02:57:24 | mriedem | https://bugs.launchpad.net/nova/+bug/1718295 | |
| 02:57:26 | openstack | Launchpad bug 1718295 in OpenStack Compute (nova) "Live migration fails with qemu-img >= 2.10: "Failed to get shared "write" lock\nIs another process using the image?"" [High,Fix released] - Assigned to Sean Dague (sdague) | |
| 02:57:48 | mriedem | so now our support matrix is likely something like, | |
| 02:57:55 | mriedem | 1. do it the old way if qemu<2.10, else | |
| 02:58:05 | mriedem | 2. do it the new way if qemu>=2.10 AND libvirt>=3.9.0 | |