| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-04 | |||
| 01:04:37 | jroll | we should make that better so it isn't expected behavior, but low priority I guess | |
| 01:04:43 | mriedem | oh | |
| 01:04:44 | mriedem | well then | |
| 01:07:00 | jroll | oh, I can't set importance ¯\_(ツ)_/¯ | |
| 01:15:56 | mriedem | join the bug team | |
| 01:15:58 | mriedem | should be able to then | |
| 01:17:25 | jroll | idk, you might make me do stuff | |
| 01:26:51 | bjhuangr | mriedem, hi, do you have a chance to review https://review.openstack.org/#/c/523387/ ? Thanks in advance . | |
| 01:33:37 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: [libvirt] Add _get_XXXpin_cpuset() https://review.openstack.org/527631 | |
| 01:33:38 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: disable cpu pinning with libvirt/qemu driver https://review.openstack.org/531049 | |
| 01:33:38 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add NumaTopology support for libvirt/qemu driver https://review.openstack.org/530451 | |
| 02:20:58 | mriedem | kashyap: could use some help with this if you get a chance https://review.openstack.org/#/c/267587/75/nova/virt/libvirt/guest.py - trying to get multiattach working which used to be ok before qemu 2.10 but now we hit issues with a write lock when attaching the volume to the 2nd guest, and i thought we could pass the force flag to the attach device call to libvirt but i got this error: | |
| 02:21:04 | mriedem | libvirtError: unsupported flags (0x4) in function qemuDomainAttachDeviceLiveAndConfig | |
| 02:25:11 | Kevin_Zheng | seems Nova is broken after add uuid to BDM | |
| 02:25:16 | Kevin_Zheng | due to https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L497 | |
| 02:25:31 | openstackgerrit | Lance Bragstad proposed openstack/nova master: Simplify logic in get_enforcer https://review.openstack.org/531008 | |
| 02:25:48 | Kevin_Zheng | the field could not be added to cell1 db | |
| 02:27:05 | mriedem | Kevin_Zheng: that nova-manage code is really old | |
| 02:27:17 | mriedem | if the bdm uuid change broke that, we should have seen it in CI | |
| 02:27:19 | mriedem | since devstack runs this | |
| 02:27:38 | mriedem | the cell1 sync happens here https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L513 | |
| 02:27:45 | mriedem | after running the schema migrations for cell0 | |
| 02:28:46 | Kevin_Zheng | Hmm, but my db didn't got updated | |
| 02:28:53 | mriedem | is your nova.conf correct? | |
| 02:30:34 | Kevin_Zheng | I will check, but I have been using this for sometime, it should be correct. | |
| 02:32:11 | Kevin_Zheng | Hmm... in config file for api service, the database connection should be cell0 db, correct? | |
| 02:35:52 | mriedem | that's what we have in http://logs.openstack.org/58/526258/3/check/tempest-full/d133d1f/controller/logs/etc/nova/ | |
| 02:36:04 | mriedem | but devstack will sync using the cell1 conf too | |
| 02:36:11 | mriedem | which has the cell1 database in the [database] section | |
| 02:36:48 | mriedem | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L707 | |
| 02:37:08 | mriedem | this runs it for cell0 https://github.com/openstack-dev/devstack/blob/master/lib/nova#L711 | |
| 02:37:29 | Kevin_Zheng | ah, I see | |
| 02:37:48 | mriedem | this was never implemented to hit all cells https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L479 | |
| 02:37:53 | mriedem | i had a patch for it but it must be abandoned | |
| 02:38:22 | mriedem | https://review.openstack.org/#/c/420973/ | |
| 02:39:26 | Kevin_Zheng | yeah, | |
| 02:39:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix up formatting for deprecate-api-extensions-policies release note https://review.openstack.org/531061 | |
| 02:39:40 | Kevin_Zheng | Thanks, problem solved | |
| 02:39:53 | Kevin_Zheng | did we mentioned this in any docs? | |
| 02:41:10 | mriedem | it should be in the install guide | |
| 02:42:22 | mriedem | oh, well, https://docs.openstack.org/nova/latest/install/controller-install-ubuntu.html#install-and-configure-components | |
| 02:42:25 | melwitt | mriedem: re: that multiattach thing, did you see this bug? https://bugzilla.redhat.com/show_bug.cgi?id=1378242 based on that it looks like there needs to be share-rw=on property set under |
|
| 02:42:26 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:42:40 | mriedem | Kevin_Zheng: that install guide works because it's configuring nova.conf to set the [database] to the cell1 db | |
| 02:42:56 | mriedem | Kevin_Zheng: that install guide was written before the superconductor mode stuff that dansmith did in devstack | |
| 02:43:04 | mriedem | where the controllers are pointed at cell0 | |
| 02:43:22 | mriedem | Kevin_Zheng: we also have https://docs.openstack.org/nova/latest/user/cells.html#setup-of-cells-v2 | |
| 02:44:12 | mriedem | it's not very clear, but there is a note in there too | |
| 02:44:13 | mriedem | "At this point, the API database can now find the cell database, and further commands will attempt to look inside. If this is a completely fresh database (such as if you’re adding a cell, or if this is a new deployment), then you will need to run nova-manage db sync on it to initialize the schema." | |
| 02:44:31 | mriedem | Kevin_Zheng: might be a good FAQs entry https://docs.openstack.org/nova/latest/user/cells.html#faqs | |
| 02:44:43 | mriedem | melwitt: nope never seen that | |
| 02:44:59 | Kevin_Zheng | yeah, | |
| 02:45:07 | Kevin_Zheng | I will add it later | |
| 02:45:40 | mriedem | melwitt: not mentioned in the domain xml docs at all https://libvirt.org/formatdomain.html | |
| 02:45:41 | mriedem | :( | |
| 02:47:27 | mriedem | and by the looks of when this was 'fixed' i'm guessing we'd need super modern versions of qemu to use this | |
| 02:47:45 | melwitt | yeah, I was just thinking the same | |
| 02:48:15 | mriedem | well, so much for multiattach in queens | |
| 02:48:21 | melwitt | this all looks fairly recent, which would explain the absence in the docs | |
| 02:48:22 | mriedem | wah wah | |
| 02:48:34 | mriedem | yeah | |
| 02:48:35 | mriedem | commit 860a3c4bea1d24773d8a495f213d5de3ac48a462 Author: Peter Krempa <pkrempa@redhat.com> Date: Wed Nov 15 15:02:58 2017 +0100 | |
| 02:50:22 | mriedem | we could have conditional logic around the version of qemu being used, but that sucks, | |
| 02:50:36 | mriedem | and i don't know if like, qemu < 2.10 works, and then is broken until qemu >= x | |
| 02:50:47 | mriedem | so you'd have a middle ground where things just don't work | |
| 02:53:08 | melwitt | what do you mean, like if this was a regression there could be a window in the middle where things don't work? | |
| 02:53:13 | mriedem | yeah | |
| 02:53:27 | mriedem | we didn't have this problem around ~newton when i wrote the original tempest test for multiattach | |
| 02:53:40 | mriedem | you could do the 2 attachments fine, it's just that nova didn't orchestrate the detach properly | |
| 02:53:45 | melwitt | oh :( | |
| 02:53:48 | mriedem | now we're in a case with newer qemu where the 2nd attach fails | |
| 02:54:07 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: log test: use fixtures.StandardLogging in setUp https://review.openstack.org/531065 | |
| 02:54:29 | mriedem | so apparently fixed in libvirt-3.9.0-3.el7 | |
| 02:54:53 | mriedem | we are testing against 3.6.0 | |
| 02:55:34 | mriedem | https://bugzilla.redhat.com/show_bug.cgi?id=1378242#c13 | |
| 02:55:35 | openstack | bugzilla.redhat.com bug 1378242 in libvirt "QEMU image file locking (libvirt)" [Unspecified,On_qa] - Assigned to pkrempa | |
| 02:55:38 | mriedem | same thing we're hitting | |
| 02:55:38 | openstack | bugzilla.redhat.com bug 1378241 in qemu-kvm-rhev "QEMU image file locking" [Unspecified,Verified] - Assigned to famz | |
| 02:55:38 | melwitt | and this is the qemu bug that the libvirt bug was cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1378241 | |
| 02:56:21 | melwitt | that says qemu-kvm-rhev though. | |
| 02:57:11 | mriedem | yeah so that caused the bug we had with 2.10 where we were hitting the same issue with qemu-img info | |
| 02:57:17 | mriedem | and had to start using the force flag for that | |
| 02:57:24 | mriedem | https://bugs.launchpad.net/nova/+bug/1718295 | |
| 02:57:26 | openstack | Launchpad bug 1718295 in OpenStack Compute (nova) "Live migration fails with qemu-img >= 2.10: "Failed to get shared "write" lock\nIs another process using the image?"" [High,Fix released] - Assigned to Sean Dague (sdague) | |
| 02:57:48 | mriedem | so now our support matrix is likely something like, | |
| 02:57:55 | mriedem | 1. do it the old way if qemu<2.10, else | |
| 02:58:05 | mriedem | 2. do it the new way if qemu>=2.10 AND libvirt>=3.9.0 | |
| 02:58:09 | mriedem | else | |
| 02:58:12 | mriedem | 3. shart your pants | |
| 02:58:31 | melwitt | sigh, so this has some history | |
| 02:58:31 | mriedem | and only 3 is testable in our current CI env :( | |
| 03:01:14 | clarkb | didnt nova implement its own lock wrapper for that? | |
| 03:01:29 | mriedem | for what? | |
| 03:01:33 | openstackgerrit | Eli Qiao proposed openstack/nova master: Api-guide: Add Block Device Mapping https://review.openstack.org/522084 | |
| 03:03:28 | clarkb | mriedem: for the lack of working locks | |
| 03:03:40 | clarkb | there wa sa while wrapper script thing iirc | |
| 03:04:42 | bjhuangr | mriedem: hi Matt, do you have a chance to review https://review.openstack.org/#/c/523387/ ? Thanks in advance | |
| 03:05:45 | mriedem | bjhuangr: not right now sorry | |
| 03:05:51 | mriedem | clarkb: sounds like something different | |
| 03:06:04 | mriedem | melwitt: looking at https://github.com/libvirt/libvirt/commit/28907b0043fbf71085a798372ab9c816ba043b93 it actually looks like that went into libvirt 3.10 | |