| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-12 | |||
| 13:07:59 | mriedem | we just pick a node randomly from those available for the host | |
| 13:08:26 | mriedem | probably need to make a note of that in the API, and another reason why the force flag is bad | |
| 13:08:40 | gibi | could we be busted here as well? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2787 | |
| 13:08:53 | gibi | the _get_compute_info call also uses the compat call to the db | |
| 13:09:47 | mriedem | possibly | |
| 13:10:54 | mriedem | gibi: well, that's if you force | |
| 13:10:58 | mriedem | if you force, scheduled_node is None | |
| 13:10:58 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2762 | |
| 13:11:22 | mriedem | so another reason why forced host evacuate won't work for ironic or vmware | |
| 13:11:30 | mriedem | although does vmware still have multiple nodes? i can't remember | |
| 13:11:36 | mriedem | bauzas: ^? | |
| 13:11:43 | gibi | ahh OK | |
| 13:13:23 | mriedem | dims: remember long ago the thing you removed for the vmware driver in nova - was that for a single compute managing multiple clusters? or something to do with a 1:M host:node mapping for vmware? | |
| 13:13:34 | mriedem | i think it was the former | |
| 13:18:09 | openstackgerrit | priyaduggirala proposed openstack/nova master: Rename parameters in call() of nova/image/glance.py https://review.openstack.org/508533 | |
| 13:21:12 | kaisers | Hi! If somebody with libvirt/assisted volume snapshot deletion experience could visit https://bugs.launchpad.net/nova/+bug/1716920 it'd be great, I'm stuck analyzing this. | |
| 13:21:13 | openstack | Launchpad bug 1716920 in OpenStack Compute (nova) "online snapshot deletion breaks volume info and backing chain (with remotefs drivers?)" [Undecided,New] | |
| 13:21:22 | kaisers | kashyap: ----^ | |
| 13:21:54 | kashyap | kaisers: Hey, I noticed your latest update on the bug | |
| 13:22:13 | kashyap | kaisers: Wasn't ignoring it; just was looking at something more broken elsewhere | |
| 13:22:36 | kaisers | kashyap: no worries :) | |
| 13:22:38 | kashyap | kaisers: You say you're back to "square one" | |
| 13:23:01 | kaisers | I believed novas block rebase operation to be stuck but that was wrong | |
| 13:23:05 | kashyap | From your comment#6, what did you patch there while testing? | |
| 13:23:26 | kaisers | kashyap: debug log messages to see what's happening and if that operation is terminating | |
| 13:23:37 | kaisers | it does | |
| 13:23:49 | kaisers | I'm not sure if i read the libvirt log correctly | |
| 13:24:06 | kaisers | (comment #7) | |
| 13:27:45 | kashyap | kaisers: I'll look in a bit; I have 2 calls to attend now. | |
| 13:27:46 | kaisers | This version of the log excerpt is probably easier to read: http://paste.openstack.org/show/623456/ | |
| 13:27:58 | kaisers | kashyap: yeah, thanks | |
| 13:27:59 | kashyap | kaisers: But here's some past notes of what to make of libvirt debug logs: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html | |
| 13:28:16 | kaisers | kashyap: will read | |
| 13:28:34 | kashyap | If you're checking request / response; for each item (request _and_ response) libvirt will increment the ID | |
| 13:29:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: add warning about force evacuate for ironic/vmware https://review.openstack.org/511477 | |
| 13:29:06 | mriedem | gibi: ^ | |
| 13:29:36 | kashyap | kaisers: Err, scratch that last line: request & corresponding response have the _same_ ID :-) Have to refer to my own notes in the e-mail | |
| 13:30:07 | kaisers | kashyap: :) | |
| 13:30:48 | dims | mriedem : this one? https://review.openstack.org/#/c/163426/ | |
| 13:30:54 | mriedem | johnthetubaguy: want to hit this pike backport? https://review.openstack.org/#/c/510957/ | |
| 13:31:06 | dims | mriedem : "map one nova-compute to one VC cluster" | |
| 13:31:08 | mriedem | dims: yeah | |
| 13:31:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 13:32:22 | dims | mriedem : we switched to one nova-compute == one cluster in vCenter. earlier you could just run a single nova-compute for all the clusters in vcenter | |
| 13:33:20 | mriedem | ok, which meant, i think, that we had a host:node ratio of 1:M | |
| 13:33:36 | mriedem | i've added rado to https://review.openstack.org/#/c/511477/ to keep me honest | |
| 13:34:08 | kaisers | kashyap: my core question currently is if i read correctly that lines 34-36 (in the paste) mean the backing file of drive-virtio-disk1 is set to volume-669cfd0c-5b5c-470b-8b25-4c15cc655373.a755e1fc-f5ff-4dc1-937f-c03fb5ecc2d8 . That would be correct and what i expect. If true it means that the result is either not written to disk for some reason or the backing file reference is changed back to it's previous value due to some follow u | |
| 13:34:44 | kaisers | kashyap: but sorry for bugging on while you're busy elsewhere, 'later | |
| 13:34:57 | mriedem | dims: ah yeah get_available_nodes only returns a single item now https://review.openstack.org/#/c/103916/13/nova/virt/vmwareapi/driver.py@389 | |
| 13:35:00 | mriedem | since that change | |
| 13:35:27 | kashyap | kaisers: No, it's okay; I see you're adding details to the bug | |
| 13:36:10 | kashyap | Ah-ha, it's with remotefs! | |
| 13:36:27 | kashyap | kaisers: You also see it with NFS? | |
| 13:36:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: add warning about force evacuate for ironic https://review.openstack.org/511477 | |
| 13:36:49 | mriedem | dims: maybe mogan will let you specify a host and node when forcing the evacuate of a baremetal instance | |
| 13:36:50 | mriedem | :) | |
| 13:37:24 | dims | 🔥:) | |
| 13:37:53 | mriedem | andreas_s: seems to be a lot of zkvm ci failures | |
| 13:38:25 | andreas_s | mriedem: thx, let me have a look | |
| 13:38:29 | mriedem | http://sng01.objectstorage.softlayer.net/v1/AUTH_1940ea10-6e82-4501-b2f9-eb236510e575/ibmzkvmci/510957/1/check-tempest-dsvm-neutron-full-ubuntu-xenial-s390x/203fd4a/console.log | |
| 13:38:35 | mriedem | ^ is a totally unrelated stable/pike change | |
| 13:39:05 | mriedem | lots of issues with volume groups and volume backups | |
| 13:39:12 | mriedem | andreas_s: is the zkvm ci also running on cinder? | |
| 13:39:21 | mriedem | because if not, you could probably exclude all of the tempest.api.volume tests | |
| 13:39:43 | mriedem | really just whitelist to running the tempest.api.compute tests | |
| 13:40:31 | kaisers | kashyap: I haven't tested this with NFS, i just noticed that e.g. vzstorage CI started skipping those tests, my guess is for a similar reason. | |
| 13:41:10 | andreas_s | mriedem: yes, it is running cinder - but sometimes the backup related tests all fail - I haven't yet figured out why this is the case.. | |
| 13:41:41 | mriedem | andreas_s: no, i mean, is the CI also running against cinder changes? | |
| 13:41:47 | mriedem | or just nova | |
| 13:41:52 | andreas_s | mriedem: ah - no. just nova | |
| 13:42:02 | mriedem | andreas_s: ok, so nova changes don't care about the results of volume api only tests | |
| 13:42:04 | kaisers | kashyap: looking at recent NFS CI test they seem to hit the same issue: http://logs.openstack.org/27/511427/2/check/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/bf61d5b/console.html | |
| 13:42:08 | mriedem | nor image or object storage or identity api tests | |
| 13:42:26 | mriedem | andreas_s: so you could just whitelist the zkvm ci, which only runs against nova changes, to only run the tempest.api.compute tests | |
| 13:42:32 | kashyap | kaisers: Okay, please write a comment in the bug with the log from NFS as plain text attachment | |
| 13:42:37 | kashyap | As these URLs will expire | |
| 13:42:40 | andreas_s | mriedem: no scenario tests? | |
| 13:43:08 | kaisers | kashyap: will do | |
| 13:43:11 | mriedem | andreas_s: you could probably run the scenario tests, but some of them don't touch nova at all i don't think, like some of the advanced networking ones | |
| 13:43:39 | mriedem | andreas_s: so maybe just whitelist tempest.api.compute and tempest.scenario for now | |
| 13:43:44 | andreas_s | mriedem: skipping storage related things will definitively help to stabilize the voting pattern... | |
| 13:43:44 | mriedem | that should help reduce some of the noise and the load | |
| 13:44:06 | andreas_s | mriedem: ok, will do, thanks for the suggestion | |
| 13:44:19 | mriedem | np, thanks for being responsive | |
| 13:49:01 | mriedem | oomichi: are schema validation errors handled separately from the expected_errors decorator in this change? https://review.openstack.org/#/c/509039/ | |
| 13:49:28 | mriedem | ah i guess it is | |
| 13:50:10 | dansmith | mriedem: so on that accepted thing, | |
| 13:50:22 | mriedem | uh oh | |
| 13:50:31 | dansmith | mriedem: I think that was improperly baking rt-level behavior into the db layer | |
| 13:50:33 | dansmith | mriedem: way before placement of course | |
| 13:50:43 | mriedem | nikola's patch? | |
| 13:50:49 | dansmith | I don't think that just because it was intentional back then means it makes sense now | |
| 13:50:51 | dansmith | yeah | |
| 13:51:05 | mriedem | i meant to go dig into the linked bug but forgot | |
| 13:52:04 | kaisers | kashyap: I've to run in a min, I'll follow up on this irc log and the bug ticket when i'm back on... | |
| 13:52:30 | kashyap | kaisers: Sure; it's best if we hash it out on the bug, too. As others can catch up on the context | |
| 13:52:41 | mriedem | dansmith: it was related to bug https://bugs.launchpad.net/nova/+bug/1417667 | |
| 13:52:42 | openstack | Launchpad bug 1417667 in OpenStack Compute (nova) "migration/evacuation/rebuild/resize of instance with NUMA topology needs to recalculate NUMA topology" [Medium,In progress] | |
| 13:52:46 | mriedem | but, that's an epic thing | |
| 13:52:48 | mriedem | so who knows | |