Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-13
15:08:45 corvus mriedem: okay, yeah. maybe worth the effort though? :) it's a one-time cost.
15:09:59 Shilpa mriedem: thank you for clarification, earlier thought that volume-backed flavors spec and https://review.openstack.org/#/q/topic:fix-bfv-boot-resources+(status:open+OR+status:merged) has relation.
15:10:24 corvus mriedem: we did come up with some ways of maybe doing something sensible with irrelevant-files and project-templates, but it's probably going to be a while till we can implement them.
15:14:19 mriedem corvus: yeah, i've likely already spent more time complaining about this than what it would take to just fix it
15:18:50 mdbooth leakypipes superdan: re live migration non-ceph->ceph there's no fundamental reason it can't work that I can think of. As was mentioned, though, it's not tested, so it's probably broken for reasons I can't think of, which are probably fixable.
15:19:30 openstackgerrit Merged openstack/nova master: Test case: ResourceClass.normalize_name with ß https://review.openstack.org/560092
15:19:42 superdan I would assume that our driver probably tries to do smart things assuming it is shared, which might defeat other things like block migration if it doesn't account for the possibility
15:20:18 mriedem live migration should try to see if it can talk to the dest host over a shared filesystem though,
15:20:28 mriedem which if you're going ceph to non-ceph, will say you can't do that
15:20:32 mriedem so block migration would be required
15:21:12 mdbooth I think the shared test is only for shared instance directory
15:21:16 superdan right
15:21:30 mdbooth The shared block iirc is in imagebackend at the class level
15:21:33 mdbooth So super-hacky
15:21:39 openstackgerrit Merged openstack/nova master: Make ResourceClass.normalize_name handle sharp S https://review.openstack.org/560093
15:21:45 mdbooth So yeah, that could potentially break it.
15:22:27 mdbooth But fundamentally, if we wired it up correctly there's no reason you can't block migration from non-ceph to ceph, or vice versa.
15:22:39 mriedem right i think it should work,
15:22:43 mriedem but there are probably assumptions in the code
15:22:49 superdan yep and that's what I'm saying
15:22:57 superdan I bet it doesn't work.. for stupid reasons
15:23:17 mriedem this is where i'd look https://github.com/openstack/nova/blob/master/nova/virt/libvirt/migration.py#L80
15:23:27 mriedem that updates the guest xml on the source host, but updates the xml intended for the dest host
15:23:29 leakypipes well, they're going to test it. I'll let you know if/what blows up :)
15:23:40 mriedem nothing in ^ indicates it cares about non-volume disks
15:24:04 mriedem this would be the other place to look https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7562
15:24:14 mriedem run on the dest, get the info to pass back to the source to tell it how to migrate it
15:27:56 mdbooth mriedem: Yeah, that's broken for at least the first reason (disk xml isn't updated)
15:29:47 openstackgerrit Chris Friesen proposed openstack/nova stable/pike: libvirt: disconnect volume from host during detach https://review.openstack.org/560690
15:30:31 mriedem corvus: finucannot: mordred: heh now i realize the flaw in my logic - the problem isn't in defining our own py27 job for unit tests,
15:30:40 mdbooth mriedem: This came up today: https://review.openstack.org/#/c/533249/ tl;dr support for ip_allocation='none'. By any chance are you aware of any other existing work on this?
15:30:44 mriedem it's that the tempest-full jobs get run on unit test-only patches in nova
15:30:46 superdan kashyap: finucannot: do you guys understand the vhostuser override stuff in this? https://review.openstack.org/#/c/484997/16/nova/virt/libvirt/vif.py line 171
15:30:52 mriedem so we'd have to write our own tempest-full...
15:31:05 mriedem mdbooth: yes, carl_baldwin added that in newton
15:31:18 mriedem mdbooth: https://blueprints.launchpad.net/nova/+spec/neutron-routed-networks
15:31:18 mdbooth mriedem: That was ip_allocation='deferred'
15:31:21 mriedem oh
15:31:24 corvus mriedem: wait, no don't write your own unit test or tempest jobs
15:31:58 mdbooth mriedem: This is for when the user literally never wants an ip
15:32:09 corvus mriedem: when you attach a job to a project, you can specify the irrelevant files. so if you do that in-repo, as opposed to using the templates, you can specify your own irrelevant files.
15:32:10 cfriesen leakypipes: try it and see? :)
15:32:33 mriedem mdbooth: then they should use networks='none' with 2.37
15:32:51 leakypipes cfriesen: yeah, they are :)
15:33:03 mdbooth Can they still attach the port without an ip?
15:33:04 leakypipes will report back with my findings from the field.
15:33:49 mriedem corvus: i think nova gets tempest-full via the integrated-gate template https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L10316
15:33:51 corvus mriedem: (the complication is that andreaf would like you to please use the project template for tempest jobs for the time being so can can adjust it centrally. so until we come up with a way to support overriding the matchers of a project-template in-repo, we have to choose between local or central control)
15:34:02 mriedem right,
15:34:21 mriedem so i'm not sure if nova would have to drop using the integrated-gate template or what
15:35:34 andreaf mriedem yeah that's the conclusion we came to at the PTG
15:35:38 corvus mriedem: yes. options are: drop that template and add the jobs it has directly; or wait until i (or someone) can get around to implementing some way to alter the irrelevant-files matchers on jobs in that template in your repo (probably wont happen for weeks at least)
15:36:01 mriedem i choose not to make a choice, like geddy said
15:37:01 mriedem https://www.youtube.com/watch?v=hX9VGUaIpds
15:37:05 mriedem where are all my rush fans?!
15:37:15 mriedem cfriesen: surely you ^
15:37:52 corvus everone take the day off and listen to rush
15:38:02 jgwentworth geddy, I wonder if he speaks like an ordinary guy
15:39:15 openstack Launchpad bug 1750623 in OpenStack Compute (nova) queens "rebuild to same host with different image shouldn't check with placement" [Medium,In progress] - Assigned to Hongbin Lu (hongbin.lu)
15:39:15 leakypipes cfriesen: ? for you on https://bugs.launchpad.net/nova/+bug/1750623
15:40:13 cfriesen mriedem: I don't think you can avoid it up here. :)
15:40:27 mdbooth mriedem: Just reading the get-me-a-network stuff. I think networks='none' is different. They *do* want to attach the port to a network, it's just that the network has no subnets, will never be associated with an IP, and therefore can't have a security group (iiuc).
15:40:46 cfriesen leakypipes: yes?
15:41:04 cfriesen leakypipes: ah, checking.
15:41:29 mdbooth mriedem: Don't worry if it doesn't ring any bells, btw. Just wanted to check if you knew of any work going on in this area other than the above review.
15:46:12 cfriesen leakypipes: yep, good catch. that just opened a huge can of worms I think.
15:46:24 leakypipes cfriesen: well, shit. I didn't intend it to. :)
15:46:40 leakypipes cfriesen: you mean *more* than the can of worms that rebuild already is? ;)
15:47:07 mriedem leakypipes: we still run the new image through the image properties filter and numa topology filter
15:47:13 mriedem but we won't claim on the compute, in the case of numa
15:47:30 cfriesen mriedem: yeah, but if the numa topology changes then placement allocations need to change
15:48:05 mriedem so we probably need some filter that says you can't rebuild with a new image that has new numa topo
15:48:36 mriedem can you specify numa topo via image props? https://docs.openstack.org/python-glanceclient/latest/cli/property-keys.html
15:48:37 cfriesen mriedem: and that additional restriction would be a behaviour change
15:48:47 cfriesen mriedem: yeah, hw_numa_nodes=2
15:48:53 mriedem not really, it's a bug fix
15:49:05 mriedem huh, ok that's not in that doc...
15:49:41 mriedem or https://docs.openstack.org/nova/latest/admin/cpu-topologies.html#smp-numa-and-smt for that matter
15:49:43 cfriesen also hw_numa_cpus.1=2, hw_numa_mem.1=512
15:50:18 mriedem yeah i see it in ImageMetaProps now
15:51:01 cfriesen you can even specify pagesize in the image if the flavor specifies "any" or "large"
15:52:12 jgwentworth anyone know if this is a known gate bug? I find some hits in logstash http://logs.openstack.org/78/554078/12/gate/legacy-tempest-dsvm-multinode-live-migration/cc18c8c/logs/testr_results.html.gz
15:52:57 openstack Launchpad bug 1763761 in OpenStack Compute (nova) "CPU topologies in nova - doesn't mention numa specific image properties" [Medium,Confirmed]
15:52:57 mriedem cfriesen: if you're so inclined, you could dump other image props in https://bugs.launchpad.net/nova/+bug/1763761
15:53:19 mriedem jgwentworth: likely yes, but you have to find the qemu guest log to see what the actual failure was
15:53:29 jgwentworth ah okay
15:54:11 mriedem http://logs.openstack.org/78/554078/12/gate/legacy-tempest-dsvm-multinode-live-migration/cc18c8c/logs/subnode-2/libvirt/qemu/instance-0000000a.txt.gz
15:54:15 mriedem VQ 1 size 0x100 Guest index 0x4 inconsistent with Host index 0x5: delta 0xffff
15:54:21 superdan hmm, jackie_truong not around on irc today eh?
15:54:39 mriedem jgwentworth: http://status.openstack.org/elastic-recheck/#1761798
15:54:57 jgwentworth ahoy
15:54:58 mriedem jgwentworth: that fingerprint needs to be updated
15:55:05 mriedem it relies on VQ 0 only
15:55:14 mriedem but this has "VQ 1 size"
15:55:20 mriedem jgwentworth: can you push an update to the query in e-r?
15:55:25 jgwentworth mriedem: yis
15:55:47 mriedem http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22VQ%5C%22%20AND%20message%3A%5C%22size%5C%22%20AND%20message%3A%5C%22Guest%20index%5C%22%20AND%20message%3A%5C%22inconsistent%20with%20Host%20index%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A1&from=7d
15:55:56 mriedem that should handle it
15:57:06 mriedem cfriesen: leakypipes: so yeah, sounds like we should fail rebuild if you attempt to use a new image that has any different numa/cpu/hw stuff in it
15:57:15 jgwentworth thankee

Earlier   Later