| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-12 | |||
| 12:54:13 | mriedem | almost wonder if we should delete those files in ocata | |
| 12:56:13 | openstackgerrit | Merged openstack/nova stable/pike: Support qemu >= 2.10 https://review.openstack.org/509774 | |
| 12:56:15 | bhagyashris | johnthetubaguy: i am not clear about how i will skip the image creation for new api version because code to skip image creation is at /nova/compute/api/py and that is common code | |
| 12:56:28 | mriedem | bhagyashris: pass a flag | |
| 12:57:00 | bhagyashris | mriedem: ohh ok | |
| 12:57:25 | mriedem | check the microversion in the REST API handler code, and pass some flag down to the compute api, with the default being backward compatible | |
| 12:57:52 | gibi | mriedem: I saw your comment and left a question about possible solution inline in https://review.openstack.org/#/c/510938/2/nova/compute/manager.py | |
| 12:59:03 | mriedem | gibi: yeah was just replying | |
| 12:59:18 | mriedem | i forgot that ironic and vmware support rebuild | |
| 13:00:17 | gibi | if it just rebuild then we don't have to care (I guess) but if they support evacuation as well then we have to me smarter | |
| 13:00:46 | mriedem | yeah i know vmware deployments use evacuate, | |
| 13:01:20 | mriedem | because before dansmith robustified it, if you started up a 2nd nova-compute to manage the same cluster as another, it would think all of the instances were evacuated and delete them from the cluster | |
| 13:01:24 | mriedem | which people didn't like for some reason | |
| 13:02:06 | mriedem | i fondly remember being on several customer calls at ibm because of that issue | |
| 13:02:10 | gibi | I can imagine the how big was that dislike :) | |
| 13:03:16 | gibi | what if I just ask for all the compute nodes for this host before the loop and look up the needed node in the loop? | |
| 13:04:56 | mriedem | gibi: replied inline | |
| 13:05:14 | mriedem | you could also probably do that | |
| 13:05:33 | mriedem | i was suggesting lookup by host/node, cache the result | |
| 13:06:07 | gibi | OK. let's do that | |
| 13:07:12 | mriedem | so that made me think of something https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L752 | |
| 13:07:30 | mriedem | if you force an evacuate for ironic or vmware, you're going to probably be busted | |
| 13:07:39 | mriedem | because you can only specify the host, but not the node | |
| 13:07:59 | mriedem | we just pick a node randomly from those available for the host | |
| 13:08:26 | mriedem | probably need to make a note of that in the API, and another reason why the force flag is bad | |
| 13:08:40 | gibi | could we be busted here as well? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2787 | |
| 13:08:53 | gibi | the _get_compute_info call also uses the compat call to the db | |
| 13:09:47 | mriedem | possibly | |
| 13:10:54 | mriedem | gibi: well, that's if you force | |
| 13:10:58 | mriedem | if you force, scheduled_node is None | |
| 13:10:58 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2762 | |
| 13:11:22 | mriedem | so another reason why forced host evacuate won't work for ironic or vmware | |
| 13:11:30 | mriedem | although does vmware still have multiple nodes? i can't remember | |
| 13:11:36 | mriedem | bauzas: ^? | |
| 13:11:43 | gibi | ahh OK | |
| 13:13:23 | mriedem | dims: remember long ago the thing you removed for the vmware driver in nova - was that for a single compute managing multiple clusters? or something to do with a 1:M host:node mapping for vmware? | |
| 13:13:34 | mriedem | i think it was the former | |
| 13:18:09 | openstackgerrit | priyaduggirala proposed openstack/nova master: Rename parameters in call() of nova/image/glance.py https://review.openstack.org/508533 | |
| 13:21:12 | kaisers | Hi! If somebody with libvirt/assisted volume snapshot deletion experience could visit https://bugs.launchpad.net/nova/+bug/1716920 it'd be great, I'm stuck analyzing this. | |
| 13:21:13 | openstack | Launchpad bug 1716920 in OpenStack Compute (nova) "online snapshot deletion breaks volume info and backing chain (with remotefs drivers?)" [Undecided,New] | |
| 13:21:22 | kaisers | kashyap: ----^ | |
| 13:21:54 | kashyap | kaisers: Hey, I noticed your latest update on the bug | |
| 13:22:13 | kashyap | kaisers: Wasn't ignoring it; just was looking at something more broken elsewhere | |
| 13:22:36 | kaisers | kashyap: no worries :) | |
| 13:22:38 | kashyap | kaisers: You say you're back to "square one" | |
| 13:23:01 | kaisers | I believed novas block rebase operation to be stuck but that was wrong | |
| 13:23:05 | kashyap | From your comment#6, what did you patch there while testing? | |
| 13:23:26 | kaisers | kashyap: debug log messages to see what's happening and if that operation is terminating | |
| 13:23:37 | kaisers | it does | |
| 13:23:49 | kaisers | I'm not sure if i read the libvirt log correctly | |
| 13:24:06 | kaisers | (comment #7) | |
| 13:27:45 | kashyap | kaisers: I'll look in a bit; I have 2 calls to attend now. | |
| 13:27:46 | kaisers | This version of the log excerpt is probably easier to read: http://paste.openstack.org/show/623456/ | |
| 13:27:58 | kaisers | kashyap: yeah, thanks | |
| 13:27:59 | kashyap | kaisers: But here's some past notes of what to make of libvirt debug logs: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html | |
| 13:28:16 | kaisers | kashyap: will read | |
| 13:28:34 | kashyap | If you're checking request / response; for each item (request _and_ response) libvirt will increment the ID | |
| 13:29:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: add warning about force evacuate for ironic/vmware https://review.openstack.org/511477 | |
| 13:29:06 | mriedem | gibi: ^ | |
| 13:29:36 | kashyap | kaisers: Err, scratch that last line: request & corresponding response have the _same_ ID :-) Have to refer to my own notes in the e-mail | |
| 13:30:07 | kaisers | kashyap: :) | |
| 13:30:48 | dims | mriedem : this one? https://review.openstack.org/#/c/163426/ | |
| 13:30:54 | mriedem | johnthetubaguy: want to hit this pike backport? https://review.openstack.org/#/c/510957/ | |
| 13:31:06 | dims | mriedem : "map one nova-compute to one VC cluster" | |
| 13:31:08 | mriedem | dims: yeah | |
| 13:31:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 13:32:22 | dims | mriedem : we switched to one nova-compute == one cluster in vCenter. earlier you could just run a single nova-compute for all the clusters in vcenter | |
| 13:33:20 | mriedem | ok, which meant, i think, that we had a host:node ratio of 1:M | |
| 13:33:36 | mriedem | i've added rado to https://review.openstack.org/#/c/511477/ to keep me honest | |
| 13:34:08 | kaisers | kashyap: my core question currently is if i read correctly that lines 34-36 (in the paste) mean the backing file of drive-virtio-disk1 is set to volume-669cfd0c-5b5c-470b-8b25-4c15cc655373.a755e1fc-f5ff-4dc1-937f-c03fb5ecc2d8 . That would be correct and what i expect. If true it means that the result is either not written to disk for some reason or the backing file reference is changed back to it's previous value due to some follow u | |
| 13:34:44 | kaisers | kashyap: but sorry for bugging on while you're busy elsewhere, 'later | |
| 13:34:57 | mriedem | dims: ah yeah get_available_nodes only returns a single item now https://review.openstack.org/#/c/103916/13/nova/virt/vmwareapi/driver.py@389 | |
| 13:35:00 | mriedem | since that change | |
| 13:35:27 | kashyap | kaisers: No, it's okay; I see you're adding details to the bug | |
| 13:36:10 | kashyap | Ah-ha, it's with remotefs! | |
| 13:36:27 | kashyap | kaisers: You also see it with NFS? | |
| 13:36:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: add warning about force evacuate for ironic https://review.openstack.org/511477 | |
| 13:36:49 | mriedem | dims: maybe mogan will let you specify a host and node when forcing the evacuate of a baremetal instance | |
| 13:36:50 | mriedem | :) | |
| 13:37:24 | dims | 🔥:) | |
| 13:37:53 | mriedem | andreas_s: seems to be a lot of zkvm ci failures | |
| 13:38:25 | andreas_s | mriedem: thx, let me have a look | |
| 13:38:29 | mriedem | http://sng01.objectstorage.softlayer.net/v1/AUTH_1940ea10-6e82-4501-b2f9-eb236510e575/ibmzkvmci/510957/1/check-tempest-dsvm-neutron-full-ubuntu-xenial-s390x/203fd4a/console.log | |
| 13:38:35 | mriedem | ^ is a totally unrelated stable/pike change | |
| 13:39:05 | mriedem | lots of issues with volume groups and volume backups | |
| 13:39:12 | mriedem | andreas_s: is the zkvm ci also running on cinder? | |
| 13:39:21 | mriedem | because if not, you could probably exclude all of the tempest.api.volume tests | |
| 13:39:43 | mriedem | really just whitelist to running the tempest.api.compute tests | |
| 13:40:31 | kaisers | kashyap: I haven't tested this with NFS, i just noticed that e.g. vzstorage CI started skipping those tests, my guess is for a similar reason. | |
| 13:41:10 | andreas_s | mriedem: yes, it is running cinder - but sometimes the backup related tests all fail - I haven't yet figured out why this is the case.. | |
| 13:41:41 | mriedem | andreas_s: no, i mean, is the CI also running against cinder changes? | |
| 13:41:47 | mriedem | or just nova | |
| 13:41:52 | andreas_s | mriedem: ah - no. just nova | |
| 13:42:02 | mriedem | andreas_s: ok, so nova changes don't care about the results of volume api only tests | |
| 13:42:04 | kaisers | kashyap: looking at recent NFS CI test they seem to hit the same issue: http://logs.openstack.org/27/511427/2/check/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/bf61d5b/console.html | |
| 13:42:08 | mriedem | nor image or object storage or identity api tests | |
| 13:42:26 | mriedem | andreas_s: so you could just whitelist the zkvm ci, which only runs against nova changes, to only run the tempest.api.compute tests | |
| 13:42:32 | kashyap | kaisers: Okay, please write a comment in the bug with the log from NFS as plain text attachment | |
| 13:42:37 | kashyap | As these URLs will expire | |
| 13:42:40 | andreas_s | mriedem: no scenario tests? | |