| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-25 | |||
| 14:12:46 | spatel | I am using raw image | |
| 14:13:13 | mdbooth | spatel: On both source and destination computes? | |
| 14:13:45 | openstackgerrit | Merged openstack/nova master: Consider allocations invovling child providers during allocation cleanup https://review.openstack.org/606050 | |
| 14:14:19 | spatel | I think yes.. i think if you move machine from A to B disk alway stay same right? | |
| 14:14:33 | mdbooth | spatel: look in nova.conf | |
| 14:14:44 | spatel | what option you want to check | |
| 14:14:49 | mdbooth | On both source and destination computes | |
| 14:15:11 | mdbooth | images_type | |
| 14:15:42 | spatel | I don't have that option in nova.conf so must be using default one | |
| 14:16:01 | spatel | is default Qcow ? | |
| 14:16:08 | mdbooth | Yes | |
| 14:16:25 | sean-k-mooney | mdbooth: if you have a raw image in glance it gets converted to qcow by defualt correct | |
| 14:16:39 | mdbooth | Actually, default is modified by use_cow config option | |
| 14:16:57 | mdbooth | spatel: Do you have use_cow set on either source or dest? | |
| 14:16:57 | sean-k-mooney | which defualt to True | |
| 14:18:13 | spatel | use_cow where? on nova.conf ( no) | |
| 14:18:31 | mdbooth | spatel: Yes. | |
| 14:18:37 | mdbooth | use_cow_images on both source and dest | |
| 14:18:43 | spatel | I don't have that option in nova.conf | |
| 14:19:06 | sean-k-mooney | ok so in that chase your vms should be using qcow images when spawned | |
| 14:19:06 | spatel | cat /etc/nova/nova.conf | grep use_cow (nothing found) | |
| 14:19:14 | mdbooth | sean-k-mooney: IIRC the flat backend can not convert qcow2 glances images if so configured, but they're always flat (no backend) | |
| 14:19:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix min config value for shutdown_timeout option https://review.openstack.org/613028 | |
| 14:19:37 | mdbooth | spatel: Are you using containers? | |
| 14:19:51 | mdbooth | spatel: i.e. are you sure you're looking in the correct nova.conf | |
| 14:20:05 | spatel | No containers | |
| 14:20:12 | sean-k-mooney | mdbooth: i think the glance iamges are raw not qcow spatel can you do an image show? | |
| 14:20:42 | sean-k-mooney | spatel: is this not an openstack stack ansible deployment. it used to default to lxc contaier no? | |
| 14:21:14 | spatel | disk_format | raw | |
| 14:21:59 | spatel | Yes i am using openstack-ansible ( it only use LXC for infra services like glance, neutron etc..) compute node run without container | |
| 14:22:36 | spatel | I am 100% sure i am looking at correct /etc/nova/nova.conf | |
| 14:23:12 | openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607 | |
| 14:23:31 | spatel | I have only one file called nova.conf | |
| 14:24:42 | sean-k-mooney | spatel: ok can you wrap this all up in a bug. the more infor you can provide the petter so if you can link an sanitised copy of your nova.conf that would help alos | |
| 14:25:02 | sean-k-mooney | e.g. remove any sensive info such as password or ips that you dont want to share | |
| 14:25:11 | spatel | I will :) | |
| 14:25:44 | spatel | Last question my glance image showing raw image but when i deploy VM it convert image to QCOW why? | |
| 14:26:07 | spatel | is that normal ? | |
| 14:28:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 14:28:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 14:28:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 14:59:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop legacy live migrate allocation compat code https://review.openstack.org/611975 | |
| 14:59:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove restart_scheduler_service() method https://review.openstack.org/613348 | |
| 15:33:31 | melwitt | ||
| 15:36:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Migrate old style volume attachments on nova-compute startup https://review.openstack.org/549130 | |
| 15:56:45 | openstackgerrit | Merged openstack/nova-specs master: Support initial allocation ratios https://review.openstack.org/552105 | |
| 16:18:05 | stephenfin | cfriesen, bauzas, artom, sean-k-mooney: So, what way are we going on https://review.openstack.org/611088 ? | |
| 16:19:08 | sean-k-mooney | stephenfin: please have a config option and i would prefer it to not be enabled by defualt but i can live with it so long as i can turn it off | |
| 16:19:22 | stephenfin | Personally, I would rather do nothing because I consider the whole thing totally broken. However, that's likely not a realistic approach. As such, it's between a config option and not using the check for forced instances | |
| 16:19:39 | stephenfin | sean-k-mooney: So default to allowing live migrations w/ NUMA or not allowing them? | |
| 16:19:50 | sean-k-mooney | default to allowing it | |
| 16:19:54 | stephenfin | Why? | |
| 16:19:57 | sean-k-mooney | mainly for ovs-dpdk | |
| 16:19:57 | cfriesen | stephenfin: default to safe, allow operators to opt in to being unsafe | |
| 16:20:18 | sean-k-mooney | all ovs-dpdk deployment have numa toplogyes because of hugepages | |
| 16:20:22 | cfriesen | opposite of sean-k-mooney :) | |
| 16:20:27 | stephenfin | cfriesen: Yeah, that's what I was thinking | |
| 16:20:58 | stephenfin | sean-k-mooney: Relatively speaking though, I imagine users with NUMA topologies >>> users with OVS-DPDK | |
| 16:21:08 | sean-k-mooney | i know yahoo japan use livemigration in there production ovs-dpdk based deployment | |
| 16:21:28 | cfriesen | sean-k-mooney: wouldn't anyone using ovs-dpdk also use pinned CPUs? | |
| 16:21:36 | sean-k-mooney | cfriesen: no | |
| 16:21:44 | cfriesen | freaky | |
| 16:21:50 | sean-k-mooney | they may but that should not be assumed | |
| 16:22:22 | sean-k-mooney | cfriesen: if you entire cloud uses ovs-dpdk you still have instance that you want to over subscive so no pinning | |
| 16:22:50 | stephenfin | sean-k-mooney: I imagine it's still fairly common though | |
| 16:23:25 | sean-k-mooney | there are 2 main usecases for ovsdpdk hosting providres that want to run allot of instnace on a singel host where the aggreate throuput exceed what kernel ovs can do | |
| 16:23:46 | sean-k-mooney | or a small number of vnf that each neeed a lot of bandwith | |
| 16:23:58 | sean-k-mooney | the frist group dont use pinning the second do | |
| 16:24:21 | stephenfin | cfriesen: So, to be clear, workaround option is good enough, as opposed to allowing it for migrations with a requested_destination? | |
| 16:24:25 | stephenfin | For you, at least | |
| 16:25:29 | stephenfin | sean-k-mooney: The way I'm thinking is that this has a potentially devastating impact on workloads and their performance, so if operators are knowingly doing it, they will switch the toggle | |
| 16:25:53 | sean-k-mooney | stephenfin: in the pinning case | |
| 16:25:53 | stephenfin | but in the meanwhile, we'll catch all the other folks who didn't know this wasn't working the way they thought | |
| 16:26:18 | sean-k-mooney | in the hugepages case the livemigation will fail if it can allocat the right hugepages | |
| 16:26:27 | sean-k-mooney | and there is no performacne impact | |
| 16:26:46 | sean-k-mooney | modulo your numa aware vswithc stuff | |
| 16:26:47 | stephenfin | can or can't? | |
| 16:27:34 | sean-k-mooney | the migration will fail if it can not alloate hugepages | |
| 16:27:47 | cfriesen | stephenfin: yes, workaround option would be good enough for me. Let's keep it simple for now. | |
| 16:27:58 | sean-k-mooney | the dest node will emit a qemu error and we will abort the migration | |
| 16:28:09 | cfriesen | I'd rather see live migrations get fixed than get fancy with the workaround. | |
| 16:28:25 | sean-k-mooney | cfriesen: same | |
| 16:28:32 | stephenfin | cfriesen: Agreed. With any luck, this won't even survive the release | |
| 16:28:39 | stephenfin | But it'll be a useful backport, IMO | |
| 16:28:54 | stephenfin | sean-k-mooney: There's also the non-homogeneous hosts case to consider too | |
| 16:28:59 | sean-k-mooney | stephenfin: if you backport it it needs to be off by defualt | |
| 16:29:09 | sean-k-mooney | stephenfin: yes i know | |
| 16:29:10 | stephenfin | sean-k-mooney: Yup, in that case, agreed | |
| 16:29:57 | stephenfin | cfriesen: Cool. I'll wait to see if artom or bauzas want to pipe up. If not, I'll rework accordingly | |
| 16:30:09 | sean-k-mooney | basically want yahoo were doing is if the migration aported (that will happen in the non-homogeneous case too) then they did a manual migration specifying the host | |
| 16:30:33 | sean-k-mooney | e.g. if it did not jsut work the found a node that could fit it and forced the migration by setting it | |
| 16:31:31 | stephenfin | It wouldn't fail if the guest ended up split over multiple host NUMA nodes though | |
| 16:31:41 | sean-k-mooney | stephenfin: you can proably simply this a lot but doing the check at the api level and if the vm has a numa toplogy just reject the migration based on the config value | |
| 16:32:27 | stephenfin | I know that brings us back into performance issue territory but still, just shows how this is all sorts of broken rn | |
| 16:32:29 | sean-k-mooney | stephenfin: you are thinkg of pinning agin yes. and ture | |
| 16:33:17 | stephenfin | No, pinning isn't needed. We restrict an instance NUMA node to a range of host cores corresponding to host NUMA node | |
| 16:33:19 | cfriesen | sean-k-mooney: do you have any thoughts on whether we should support TIS vs CRB in the emulated TPM spec? and if so, should that be a separate thing from the TPM version? | |
| 16:33:24 | sean-k-mooney | stephenfin: yes it is but we know that you are technically in undefined behavior land if you migrate with a numa toplogy | |
| 16:33:52 | sean-k-mooney | cfriesen: i know thate they are a thing but not enough to comment on that | |
| 16:34:18 | stephenfin | Also, mriedem noted about doing it higher but it seemed no simpler. This doesn't affect anything but libvirt so that still seems like the place to do this. | |