| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-13 | |||
| 15:40:46 | cfriesen | leakypipes: yes? | |
| 15:41:04 | cfriesen | leakypipes: ah, checking. | |
| 15:41:29 | mdbooth | mriedem: Don't worry if it doesn't ring any bells, btw. Just wanted to check if you knew of any work going on in this area other than the above review. | |
| 15:46:12 | cfriesen | leakypipes: yep, good catch. that just opened a huge can of worms I think. | |
| 15:46:24 | leakypipes | cfriesen: well, shit. I didn't intend it to. :) | |
| 15:46:40 | leakypipes | cfriesen: you mean *more* than the can of worms that rebuild already is? ;) | |
| 15:47:07 | mriedem | leakypipes: we still run the new image through the image properties filter and numa topology filter | |
| 15:47:13 | mriedem | but we won't claim on the compute, in the case of numa | |
| 15:47:30 | cfriesen | mriedem: yeah, but if the numa topology changes then placement allocations need to change | |
| 15:48:05 | mriedem | so we probably need some filter that says you can't rebuild with a new image that has new numa topo | |
| 15:48:36 | mriedem | can you specify numa topo via image props? https://docs.openstack.org/python-glanceclient/latest/cli/property-keys.html | |
| 15:48:37 | cfriesen | mriedem: and that additional restriction would be a behaviour change | |
| 15:48:47 | cfriesen | mriedem: yeah, hw_numa_nodes=2 | |
| 15:48:53 | mriedem | not really, it's a bug fix | |
| 15:49:05 | mriedem | huh, ok that's not in that doc... | |
| 15:49:41 | mriedem | or https://docs.openstack.org/nova/latest/admin/cpu-topologies.html#smp-numa-and-smt for that matter | |
| 15:49:43 | cfriesen | also hw_numa_cpus.1=2, hw_numa_mem.1=512 | |
| 15:50:18 | mriedem | yeah i see it in ImageMetaProps now | |
| 15:51:01 | cfriesen | you can even specify pagesize in the image if the flavor specifies "any" or "large" | |
| 15:52:12 | jgwentworth | anyone know if this is a known gate bug? I find some hits in logstash http://logs.openstack.org/78/554078/12/gate/legacy-tempest-dsvm-multinode-live-migration/cc18c8c/logs/testr_results.html.gz | |
| 15:52:57 | mriedem | cfriesen: if you're so inclined, you could dump other image props in https://bugs.launchpad.net/nova/+bug/1763761 | |
| 15:52:57 | openstack | Launchpad bug 1763761 in OpenStack Compute (nova) "CPU topologies in nova - doesn't mention numa specific image properties" [Medium,Confirmed] | |
| 15:53:19 | mriedem | jgwentworth: likely yes, but you have to find the qemu guest log to see what the actual failure was | |
| 15:53:29 | jgwentworth | ah okay | |
| 15:54:11 | mriedem | http://logs.openstack.org/78/554078/12/gate/legacy-tempest-dsvm-multinode-live-migration/cc18c8c/logs/subnode-2/libvirt/qemu/instance-0000000a.txt.gz | |
| 15:54:15 | mriedem | VQ 1 size 0x100 Guest index 0x4 inconsistent with Host index 0x5: delta 0xffff | |
| 15:54:21 | superdan | hmm, jackie_truong not around on irc today eh? | |
| 15:54:39 | mriedem | jgwentworth: http://status.openstack.org/elastic-recheck/#1761798 | |
| 15:54:57 | jgwentworth | ahoy | |
| 15:54:58 | mriedem | jgwentworth: that fingerprint needs to be updated | |
| 15:55:05 | mriedem | it relies on VQ 0 only | |
| 15:55:14 | mriedem | but this has "VQ 1 size" | |
| 15:55:20 | mriedem | jgwentworth: can you push an update to the query in e-r? | |
| 15:55:25 | jgwentworth | mriedem: yis | |
| 15:55:47 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22VQ%5C%22%20AND%20message%3A%5C%22size%5C%22%20AND%20message%3A%5C%22Guest%20index%5C%22%20AND%20message%3A%5C%22inconsistent%20with%20Host%20index%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A1&from=7d | |
| 15:55:56 | mriedem | that should handle it | |
| 15:57:06 | mriedem | cfriesen: leakypipes: so yeah, sounds like we should fail rebuild if you attempt to use a new image that has any different numa/cpu/hw stuff in it | |
| 15:57:15 | jgwentworth | thankee | |
| 15:57:17 | mriedem | which would be a bug since....juno? | |
| 15:59:38 | mriedem | johnthetubaguy_: i added you to https://review.openstack.org/#/c/552078/ in case you can give any feedback before i rev it | |
| 16:00:34 | johnthetubaguy_ | mriedem: will try take a peak at that, been out at a conference-ey thing all week, so miles behind the curve right now | |
| 16:01:28 | mriedem | ack thanks | |
| 16:01:56 | cfriesen | mriedem: I think the issue is more that we don't recalculate the numa topology on a rebuild....we actually used to do a claim on a rebuild | |
| 16:02:04 | mriedem | johnthetubaguy_: were these guys there? https://i.ytimg.com/vi/sVw0i5Rd_Is/maxresdefault.jpg | |
| 16:02:22 | mriedem | cfriesen: when did we ever do a claim on a rebuild? | |
| 16:02:29 | mriedem | and don't confuse rebuild with evacuate | |
| 16:02:49 | cfriesen | mriedem: yeah, you're right it was for the evac case. | |
| 16:02:49 | johnthetubaguy_ | mriedem: I hadn't noticed the resemblance till just now, but yes they were | |
| 16:03:32 | johnthetubaguy_ | mriedem: although it probably makes me scooter | |
| 16:04:02 | mriedem | ha | |
| 16:04:11 | jgwentworth | mriedem: https://review.openstack.org/#/c/561261 | |
| 16:04:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 16:04:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 16:04:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Plumb trusted_certs through libvirt driver image paths https://review.openstack.org/561262 | |
| 16:04:39 | cfriesen | mriedem: you going to raise a bug report for the image validation on rebuild? | |
| 16:04:48 | mriedem | cfriesen: no, can you please? | |
| 16:04:51 | cfriesen | sure | |
| 16:04:55 | mriedem | thanks | |
| 16:06:00 | jgwentworth | superdan, mriedem, leakypipes, alex_xu, johnthetubaguy_, finucannot, bauzas, fried_rice: been meaning to ask for review on the rocky priorities doc page https://review.openstack.org/#/c/557796 | |
| 16:06:51 | mriedem | ack | |
| 16:07:05 | fried_rice | jgwentworth: There was a typo it looked like Mel wanted to fix. Otherwise I'm +1 (discussion inline leading to possible external actions, not in the patch) | |
| 16:07:23 | fried_rice | jgwentworth: ...a typo and a link | |
| 16:07:50 | jgwentworth | oh, right. I forgot | |
| 16:08:11 | jgwentworth | I don't think there's a way to filter only Approved blueprints in launchpad, which sucks | |
| 16:08:26 | jgwentworth | for linking | |
| 16:09:29 | fried_rice | jgwentworth: There is some mystical querystring syntax that only mriedem knows how to use. | |
| 16:10:10 | jgwentworth | I thought I've asked him about that before. maybe he's holding out on me | |
| 16:10:56 | mriedem | not in the UI that i know of, | |
| 16:10:59 | mriedem | via the REST API yes | |
| 16:11:09 | superdan | I think LP has a bunch of inaccessible query strings you can use | |
| 16:11:13 | mriedem | right | |
| 16:11:32 | mriedem | https://github.com/openstack/nova-specs/blob/master/tools/move_implemented_specs.py | |
| 16:12:40 | sahid | superdan: i saw your comments and questioning on the tx/rx queue size patch. I tried to give more details I hope that is going to help | |
| 16:15:31 | superdan | sahid: okay the thing I'm worried about is I thought that the guest had to support vhost (or is it vhostuser?) in a specific way, and I haven't gone to chase down all the places where we choose between those different types of virtio backends | |
| 16:15:31 | jgwentworth | mriedem: cool script | |
| 16:18:22 | cfriesen | mriedem: here's the bug: | |
| 16:18:23 | cfriesen | LOG.info("Device %s not found in instance.", | |
| 16:18:25 | cfriesen | disk_dev, instance=instance | |
| 16:18:30 | cfriesen | grr......https://bugs.launchpad.net/nova/+bug/1763766 | |
| 16:18:30 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) "nova needs to disallow topology changes on image rebuild" [Undecided,New] | |
| 16:20:14 | superdan | sahid: okay re-reading this with your context I have a little better idea on what to look at | |
| 16:20:20 | sahid | superdan: the driver are on the guest OS. that is your question? | |
| 16:20:23 | superdan | sahid: I'll try to do that while you wor on your unit test | |
| 16:20:32 | superdan | sahid: yeah, the guest | |
| 16:21:02 | superdan | sahid: I thought that if we specify one of those (I think vhostuser) the guest _has_ to have something new enough to support it? | |
| 16:21:15 | superdan | although maybe that doesn't matter since you're only overriding back to vhost/ | |
| 16:22:51 | superdan | sahid: I guess it would be less confusing if you did your refactor of get_base_config() first | |
| 16:23:09 | mriedem | jgwentworth: one thing i've always meant to do with that is move the validation that the spec filename matches the blueprint in LP into the doc/test validation code that runs on each patch | |
| 16:23:17 | sahid | well for vhostuser the virtio driver needs to be recent enough and for vhost the kernel recent enough | |
| 16:23:18 | superdan | sahid: can/do we validate some of these types in upstream CI? | |
| 16:23:19 | mriedem | since people frequently make that mistake and then we have to fix it up later | |
| 16:24:01 | jgwentworth | mriedem: you mean in the tox run of nova-specs | |
| 16:24:06 | mriedem | yeah | |
| 16:24:08 | sahid | superdan: yes probably i was thinking about to refactor that method first... | |
| 16:24:08 | superdan | sahid: yeah, I'm just trying to make sure we're not ever selecting something different after this patch, or at least, selecting something different than what would work today | |
| 16:24:20 | sahid | oh i see | |
| 16:24:22 | jgwentworth | makes sense | |
| 16:24:26 | sahid | normally no :) | |
| 16:24:51 | superdan | sahid: if you did that first, then if we have a problem with this later, it would point to the refactor patch instead of your queues patch which would be better I think | |
| 16:25:13 | superdan | sahid: "if we had a problem" meaning.. we selected vhostuser when we should/would have selected vhost or something | |