| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-12 | |||
| 13:27:46 | kaisers | This version of the log excerpt is probably easier to read: http://paste.openstack.org/show/623456/ | |
| 13:27:58 | kaisers | kashyap: yeah, thanks | |
| 13:27:59 | kashyap | kaisers: But here's some past notes of what to make of libvirt debug logs: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html | |
| 13:28:16 | kaisers | kashyap: will read | |
| 13:28:34 | kashyap | If you're checking request / response; for each item (request _and_ response) libvirt will increment the ID | |
| 13:29:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: add warning about force evacuate for ironic/vmware https://review.openstack.org/511477 | |
| 13:29:06 | mriedem | gibi: ^ | |
| 13:29:36 | kashyap | kaisers: Err, scratch that last line: request & corresponding response have the _same_ ID :-) Have to refer to my own notes in the e-mail | |
| 13:30:07 | kaisers | kashyap: :) | |
| 13:30:48 | dims | mriedem : this one? https://review.openstack.org/#/c/163426/ | |
| 13:30:54 | mriedem | johnthetubaguy: want to hit this pike backport? https://review.openstack.org/#/c/510957/ | |
| 13:31:06 | dims | mriedem : "map one nova-compute to one VC cluster" | |
| 13:31:08 | mriedem | dims: yeah | |
| 13:31:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 13:32:22 | dims | mriedem : we switched to one nova-compute == one cluster in vCenter. earlier you could just run a single nova-compute for all the clusters in vcenter | |
| 13:33:20 | mriedem | ok, which meant, i think, that we had a host:node ratio of 1:M | |
| 13:33:36 | mriedem | i've added rado to https://review.openstack.org/#/c/511477/ to keep me honest | |
| 13:34:08 | kaisers | kashyap: my core question currently is if i read correctly that lines 34-36 (in the paste) mean the backing file of drive-virtio-disk1 is set to volume-669cfd0c-5b5c-470b-8b25-4c15cc655373.a755e1fc-f5ff-4dc1-937f-c03fb5ecc2d8 . That would be correct and what i expect. If true it means that the result is either not written to disk for some reason or the backing file reference is changed back to it's previous value due to some follow u | |
| 13:34:44 | kaisers | kashyap: but sorry for bugging on while you're busy elsewhere, 'later | |
| 13:34:57 | mriedem | dims: ah yeah get_available_nodes only returns a single item now https://review.openstack.org/#/c/103916/13/nova/virt/vmwareapi/driver.py@389 | |
| 13:35:00 | mriedem | since that change | |
| 13:35:27 | kashyap | kaisers: No, it's okay; I see you're adding details to the bug | |
| 13:36:10 | kashyap | Ah-ha, it's with remotefs! | |
| 13:36:27 | kashyap | kaisers: You also see it with NFS? | |
| 13:36:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: add warning about force evacuate for ironic https://review.openstack.org/511477 | |
| 13:36:49 | mriedem | dims: maybe mogan will let you specify a host and node when forcing the evacuate of a baremetal instance | |
| 13:36:50 | mriedem | :) | |
| 13:37:24 | dims | 🔥:) | |
| 13:37:53 | mriedem | andreas_s: seems to be a lot of zkvm ci failures | |
| 13:38:25 | andreas_s | mriedem: thx, let me have a look | |
| 13:38:29 | mriedem | http://sng01.objectstorage.softlayer.net/v1/AUTH_1940ea10-6e82-4501-b2f9-eb236510e575/ibmzkvmci/510957/1/check-tempest-dsvm-neutron-full-ubuntu-xenial-s390x/203fd4a/console.log | |
| 13:38:35 | mriedem | ^ is a totally unrelated stable/pike change | |
| 13:39:05 | mriedem | lots of issues with volume groups and volume backups | |
| 13:39:12 | mriedem | andreas_s: is the zkvm ci also running on cinder? | |
| 13:39:21 | mriedem | because if not, you could probably exclude all of the tempest.api.volume tests | |
| 13:39:43 | mriedem | really just whitelist to running the tempest.api.compute tests | |
| 13:40:31 | kaisers | kashyap: I haven't tested this with NFS, i just noticed that e.g. vzstorage CI started skipping those tests, my guess is for a similar reason. | |
| 13:41:10 | andreas_s | mriedem: yes, it is running cinder - but sometimes the backup related tests all fail - I haven't yet figured out why this is the case.. | |
| 13:41:41 | mriedem | andreas_s: no, i mean, is the CI also running against cinder changes? | |
| 13:41:47 | mriedem | or just nova | |
| 13:41:52 | andreas_s | mriedem: ah - no. just nova | |
| 13:42:02 | mriedem | andreas_s: ok, so nova changes don't care about the results of volume api only tests | |
| 13:42:04 | kaisers | kashyap: looking at recent NFS CI test they seem to hit the same issue: http://logs.openstack.org/27/511427/2/check/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/bf61d5b/console.html | |
| 13:42:08 | mriedem | nor image or object storage or identity api tests | |
| 13:42:26 | mriedem | andreas_s: so you could just whitelist the zkvm ci, which only runs against nova changes, to only run the tempest.api.compute tests | |
| 13:42:32 | kashyap | kaisers: Okay, please write a comment in the bug with the log from NFS as plain text attachment | |
| 13:42:37 | kashyap | As these URLs will expire | |
| 13:42:40 | andreas_s | mriedem: no scenario tests? | |
| 13:43:08 | kaisers | kashyap: will do | |
| 13:43:11 | mriedem | andreas_s: you could probably run the scenario tests, but some of them don't touch nova at all i don't think, like some of the advanced networking ones | |
| 13:43:39 | mriedem | andreas_s: so maybe just whitelist tempest.api.compute and tempest.scenario for now | |
| 13:43:44 | mriedem | that should help reduce some of the noise and the load | |
| 13:43:44 | andreas_s | mriedem: skipping storage related things will definitively help to stabilize the voting pattern... | |
| 13:44:06 | andreas_s | mriedem: ok, will do, thanks for the suggestion | |
| 13:44:19 | mriedem | np, thanks for being responsive | |
| 13:49:01 | mriedem | oomichi: are schema validation errors handled separately from the expected_errors decorator in this change? https://review.openstack.org/#/c/509039/ | |
| 13:49:28 | mriedem | ah i guess it is | |
| 13:50:10 | dansmith | mriedem: so on that accepted thing, | |
| 13:50:22 | mriedem | uh oh | |
| 13:50:31 | dansmith | mriedem: I think that was improperly baking rt-level behavior into the db layer | |
| 13:50:33 | dansmith | mriedem: way before placement of course | |
| 13:50:43 | mriedem | nikola's patch? | |
| 13:50:49 | dansmith | I don't think that just because it was intentional back then means it makes sense now | |
| 13:50:51 | dansmith | yeah | |
| 13:51:05 | mriedem | i meant to go dig into the linked bug but forgot | |
| 13:52:04 | kaisers | kashyap: I've to run in a min, I'll follow up on this irc log and the bug ticket when i'm back on... | |
| 13:52:30 | kashyap | kaisers: Sure; it's best if we hash it out on the bug, too. As others can catch up on the context | |
| 13:52:41 | mriedem | dansmith: it was related to bug https://bugs.launchpad.net/nova/+bug/1417667 | |
| 13:52:42 | openstack | Launchpad bug 1417667 in OpenStack Compute (nova) "migration/evacuation/rebuild/resize of instance with NUMA topology needs to recalculate NUMA topology" [Medium,In progress] | |
| 13:52:46 | mriedem | but, that's an epic thing | |
| 13:52:48 | mriedem | so who knows | |
| 13:52:57 | mriedem | https://review.openstack.org/#/q/topic:bug/1417667+(status:open+OR+status:merged) | |
| 13:52:59 | mriedem | jesus | |
| 13:53:19 | dansmith | hmm. not sure about that | |
| 13:53:25 | mriedem | heh, "this numa feature we added doens't work, it's a bug, so let's fix it with a billion patches" | |
| 13:53:26 | bhagyashris | johnthetubaguy, mriedem: thank you :) | |
| 13:55:23 | efried | sdague (cc mordred) Regarding barbican affordance in bp/use-ksa-adapter-for-endpoints -- I've been looking into it, and I don't think Nova is the right place to tackle it. Would like to discuss when you have a few. | |
| 13:56:44 | efried | (mriedem ^ if it turns out to be the case that we don't do that, would a spec update be necessary/desirable?) | |
| 13:57:59 | mriedem | efried: if the spec says you're going to do it for barbican too and you don't, then amending the spec is fine | |
| 13:58:14 | efried | alex_xu I think Jay is on quasi-vacation | |
| 13:58:17 | efried | mriedem ack | |
| 13:58:42 | alex_xu | efried: ah, thanks, when he is back? | |
| 13:58:46 | mriedem | dansmith: i don't really see anything in that series of nikola's changes, anything obvious at least, that warrants the need for including 'accepted' in that not-in-progress list, so i assume it was just something he noticed while going through that stuff | |
| 13:59:01 | efried | Waidaminute... alex_xu comes back and jaypipes is gone.... Has anyone ever actually seen them in the same place at the same time? | |
| 13:59:06 | dansmith | mriedem: yeah seems unrelated | |
| 13:59:23 | mriedem | efried: i have actually | |
| 13:59:25 | efried | alex_xu Not totally sure. Couple weeks? But he's been checking in spottily | |
| 13:59:40 | efried | mriedem Yeah, I did at the PTG. Just being hilariously funny. | |
| 13:59:42 | mriedem | talking about trusted filters and traits | |
| 14:00:23 | alex_xu | efried: ok...thanks | |
| 14:00:39 | efried | alex_xu You looking for SQL brainstorm collab? | |
| 14:00:56 | efried | on traits stuff | |
| 14:01:19 | alex_xu | efried: yea, I rewrite the trait patch as Jay suggested, want to get jay take a look whether it is he expected | |
| 14:01:34 | bauzas | mriedem: sorry, was around | |
| 14:01:48 | bauzas | mriedem: so, no, VMware is no longer using more than one node | |
| 14:01:50 | efried | alex_xu Cool. I have to admit, the way you had it before was pretty baffling to me, though I'm definitely not an expert. | |
| 14:02:04 | efried | alex_xu Which patch? | |
| 14:02:13 | alex_xu | efried: here is one https://review.openstack.org/#/c/479766/10 | |
| 14:02:17 | efried | ... | |
| 14:02:51 | alex_xu | yea, the sql is complex, but as jay said, do that in the python also complex... | |