| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-25 | |||
| 13:57:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840 | |
| 13:57:04 | sean-k-mooney | am maybe i would be very surpried but that could explain things | |
| 13:57:24 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267 | |
| 13:57:29 | mdbooth | The reason being we explicitly only copy the delta, as we copy the backing file explicitly oob. | |
| 13:57:53 | mdbooth | But if the dest was raw/flat there would be no backing file. | |
| 13:57:58 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847 | |
| 13:57:59 | sean-k-mooney | mdbooth: ill ask for more info when spatel is back on line. they were going to open a bug for this anyway | |
| 13:58:08 | mdbooth | That would be NOTABUG | |
| 13:58:20 | mdbooth | We don't support that. Not that it wouldn't be nice, but we don't. | |
| 13:58:24 | sean-k-mooney | well its a bug that nova is not chekcing for this and refusing to migrate | |
| 13:58:44 | sean-k-mooney | from an api persepcitive the migration completed sucessfully | |
| 13:58:55 | sean-k-mooney | it also deleted the original disk on the souce | |
| 13:59:19 | mdbooth | Did we ever get sticky host aggregates? | |
| 13:59:42 | mdbooth | i.e. we don't care which host aggregate is selected, but whichever one it is we never go anywhere else. | |
| 13:59:44 | sean-k-mooney | e.g. can migrate out of them by defualt without force | |
| 14:00:06 | sean-k-mooney | mdbooth: no that only happens for AZs today | |
| 14:00:29 | sean-k-mooney | at least as far as i am aware | |
| 14:06:19 | spatel | back | |
| 14:07:50 | spatel | sean-k-mooney: Do you want me to file a bug on this issue | |
| 14:08:02 | spatel | so we have more eyeballs | |
| 14:09:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes https://review.openstack.org/613126 | |
| 14:10:55 | sean-k-mooney | yes please. one thing that mdbooth suggested was to check that the raw images are used on both compute hosts | |
| 14:12:18 | sean-k-mooney | spatel: if you are using qcow on the souce node and raw on the dest we think it could cause the issue you are seeing | |
| 14:12:46 | spatel | I am using raw image | |
| 14:13:13 | mdbooth | spatel: On both source and destination computes? | |
| 14:13:45 | openstackgerrit | Merged openstack/nova master: Consider allocations invovling child providers during allocation cleanup https://review.openstack.org/606050 | |
| 14:14:19 | spatel | I think yes.. i think if you move machine from A to B disk alway stay same right? | |
| 14:14:33 | mdbooth | spatel: look in nova.conf | |
| 14:14:44 | spatel | what option you want to check | |
| 14:14:49 | mdbooth | On both source and destination computes | |
| 14:15:11 | mdbooth | images_type | |
| 14:15:42 | spatel | I don't have that option in nova.conf so must be using default one | |
| 14:16:01 | spatel | is default Qcow ? | |
| 14:16:08 | mdbooth | Yes | |
| 14:16:25 | sean-k-mooney | mdbooth: if you have a raw image in glance it gets converted to qcow by defualt correct | |
| 14:16:39 | mdbooth | Actually, default is modified by use_cow config option | |
| 14:16:57 | sean-k-mooney | which defualt to True | |
| 14:16:57 | mdbooth | spatel: Do you have use_cow set on either source or dest? | |
| 14:18:13 | spatel | use_cow where? on nova.conf ( no) | |
| 14:18:31 | mdbooth | spatel: Yes. | |
| 14:18:37 | mdbooth | use_cow_images on both source and dest | |
| 14:18:43 | spatel | I don't have that option in nova.conf | |
| 14:19:06 | spatel | cat /etc/nova/nova.conf | grep use_cow (nothing found) | |
| 14:19:06 | sean-k-mooney | ok so in that chase your vms should be using qcow images when spawned | |
| 14:19:14 | mdbooth | sean-k-mooney: IIRC the flat backend can not convert qcow2 glances images if so configured, but they're always flat (no backend) | |
| 14:19:31 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix min config value for shutdown_timeout option https://review.openstack.org/613028 | |
| 14:19:37 | mdbooth | spatel: Are you using containers? | |
| 14:19:51 | mdbooth | spatel: i.e. are you sure you're looking in the correct nova.conf | |
| 14:20:05 | spatel | No containers | |
| 14:20:12 | sean-k-mooney | mdbooth: i think the glance iamges are raw not qcow spatel can you do an image show? | |
| 14:20:42 | sean-k-mooney | spatel: is this not an openstack stack ansible deployment. it used to default to lxc contaier no? | |
| 14:21:14 | spatel | disk_format | raw | |
| 14:21:59 | spatel | Yes i am using openstack-ansible ( it only use LXC for infra services like glance, neutron etc..) compute node run without container | |
| 14:22:36 | spatel | I am 100% sure i am looking at correct /etc/nova/nova.conf | |
| 14:23:12 | openstackgerrit | Maciej Jozefczyk proposed openstack/nova master: Force refresh instance info_cache during heal https://review.openstack.org/591607 | |
| 14:23:31 | spatel | I have only one file called nova.conf | |
| 14:24:42 | sean-k-mooney | spatel: ok can you wrap this all up in a bug. the more infor you can provide the petter so if you can link an sanitised copy of your nova.conf that would help alos | |
| 14:25:02 | sean-k-mooney | e.g. remove any sensive info such as password or ips that you dont want to share | |
| 14:25:11 | spatel | I will :) | |
| 14:25:44 | spatel | Last question my glance image showing raw image but when i deploy VM it convert image to QCOW why? | |
| 14:26:07 | spatel | is that normal ? | |
| 14:28:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send resource allocations in the port binding https://review.openstack.org/569459 | |
| 14:28:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 14:28:21 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 14:59:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove restart_scheduler_service() method https://review.openstack.org/613348 | |
| 14:59:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop legacy live migrate allocation compat code https://review.openstack.org/611975 | |
| 15:33:31 | melwitt | ||
| 15:36:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Migrate old style volume attachments on nova-compute startup https://review.openstack.org/549130 | |
| 15:56:45 | openstackgerrit | Merged openstack/nova-specs master: Support initial allocation ratios https://review.openstack.org/552105 | |
| 16:18:05 | stephenfin | cfriesen, bauzas, artom, sean-k-mooney: So, what way are we going on https://review.openstack.org/611088 ? | |
| 16:19:08 | sean-k-mooney | stephenfin: please have a config option and i would prefer it to not be enabled by defualt but i can live with it so long as i can turn it off | |
| 16:19:22 | stephenfin | Personally, I would rather do nothing because I consider the whole thing totally broken. However, that's likely not a realistic approach. As such, it's between a config option and not using the check for forced instances | |
| 16:19:39 | stephenfin | sean-k-mooney: So default to allowing live migrations w/ NUMA or not allowing them? | |
| 16:19:50 | sean-k-mooney | default to allowing it | |
| 16:19:54 | stephenfin | Why? | |
| 16:19:57 | cfriesen | stephenfin: default to safe, allow operators to opt in to being unsafe | |
| 16:19:57 | sean-k-mooney | mainly for ovs-dpdk | |
| 16:20:18 | sean-k-mooney | all ovs-dpdk deployment have numa toplogyes because of hugepages | |
| 16:20:22 | cfriesen | opposite of sean-k-mooney :) | |
| 16:20:27 | stephenfin | cfriesen: Yeah, that's what I was thinking | |
| 16:20:58 | stephenfin | sean-k-mooney: Relatively speaking though, I imagine users with NUMA topologies >>> users with OVS-DPDK | |
| 16:21:08 | sean-k-mooney | i know yahoo japan use livemigration in there production ovs-dpdk based deployment | |
| 16:21:28 | cfriesen | sean-k-mooney: wouldn't anyone using ovs-dpdk also use pinned CPUs? | |
| 16:21:36 | sean-k-mooney | cfriesen: no | |
| 16:21:44 | cfriesen | freaky | |
| 16:21:50 | sean-k-mooney | they may but that should not be assumed | |
| 16:22:22 | sean-k-mooney | cfriesen: if you entire cloud uses ovs-dpdk you still have instance that you want to over subscive so no pinning | |
| 16:22:50 | stephenfin | sean-k-mooney: I imagine it's still fairly common though | |
| 16:23:25 | sean-k-mooney | there are 2 main usecases for ovsdpdk hosting providres that want to run allot of instnace on a singel host where the aggreate throuput exceed what kernel ovs can do | |
| 16:23:46 | sean-k-mooney | or a small number of vnf that each neeed a lot of bandwith | |
| 16:23:58 | sean-k-mooney | the frist group dont use pinning the second do | |
| 16:24:21 | stephenfin | cfriesen: So, to be clear, workaround option is good enough, as opposed to allowing it for migrations with a requested_destination? | |
| 16:24:25 | stephenfin | For you, at least | |
| 16:25:29 | stephenfin | sean-k-mooney: The way I'm thinking is that this has a potentially devastating impact on workloads and their performance, so if operators are knowingly doing it, they will switch the toggle | |
| 16:25:53 | stephenfin | but in the meanwhile, we'll catch all the other folks who didn't know this wasn't working the way they thought | |
| 16:25:53 | sean-k-mooney | stephenfin: in the pinning case | |
| 16:26:18 | sean-k-mooney | in the hugepages case the livemigation will fail if it can allocat the right hugepages | |
| 16:26:27 | sean-k-mooney | and there is no performacne impact | |
| 16:26:46 | sean-k-mooney | modulo your numa aware vswithc stuff | |
| 16:26:47 | stephenfin | can or can't? | |