| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-14 | |||
| 06:21:49 | Kevin_Zheng | fanzhang Are you doing it in Linux or windows | |
| 06:22:37 | fanzhang | Kevin_Zheng Linux and MacOS | |
| 06:24:11 | Kevin_Zheng | Thanks, we will try latter | |
| 06:24:45 | fanzhang | np | |
| 07:20:55 | gibi | mriedem: hi! I'm not planning to have a notification meeting also I assume your timezone is not friendly for that meeting | |
| 07:39:56 | sean-k-mooney | mriedem: o/ i have been doing some livemigration testing between different backends and different OSs and have been finding issues. im going to keep refining my testing but while it is posible to move the vms i dont think its working correctly | |
| 07:41:10 | sean-k-mooney | mriedem: live migration from cetos to ubuntu is broken. that used to work but the emltor path is nolonger the same and we use the emulator path of the src for the dest when checnking compatablity | |
| 07:41:50 | sean-k-mooney | live migration from lb to ovs appears to work but ovs to lb times out waiting for the plug event | |
| 07:41:55 | sean-k-mooney | from neutron | |
| 07:42:52 | sean-k-mooney | im going to add a few more nodes to my testing to confirm all the behavior. ill send an email to the list with a summary when done | |
| 08:01:02 | gryf | sean-k-mooney, I've answered your review. | |
| 08:01:10 | gryf | erm. comments. | |
| 08:02:19 | ygk_12345 | hi all | |
| 08:02:47 | gryf | johnthetubaguy, could you please also look at https://review.openstack.org/#/c/536351 ? | |
| 08:03:05 | ygk_12345 | what does negative number indicate for RAM in hypervisor stats ? | |
| 08:03:34 | ygk_12345 | does it mean no more memory available for vms on the computes ? | |
| 08:15:20 | sean-k-mooney | gryf: thanks, you are correct that the disk filter is deprcated rather than removed. | |
| 08:17:32 | sean-k-mooney | the disk fileter is calulating the avialble space its self form the host_state.free_diks_mb and host_state.total_usable_disk_gb | |
| 08:18:56 | sean-k-mooney | i am not sure how host_state.total_usable_disk_gb is related to the migrtion_data disk available_gb | |
| 08:19:16 | sean-k-mooney | they apepar to be independent | |
| 08:21:18 | mriedem | gibi: yeah i won't be around for the notification meeting | |
| 08:21:31 | sean-k-mooney | ygk_12345: it means that the total memory used is grater than total ram - reserved ram | |
| 08:21:33 | gibi | mriedem: no problem | |
| 08:22:05 | mriedem | sean-k-mooney: ok i think lb plug events is a known issue isn't it? | |
| 08:22:14 | ygk_12345 | sean-k-mooney: also I am getting warnings in the nova logs that enough memory cant be allocated . what does this mean ? | |
| 08:24:27 | sean-k-mooney | mriedem: well live migration between 2 lb nodes seams to work. i thikn libvirt is using the souce xml without the updated network interfaces in the ovs case and that is breaking things. i have not really had time to dig into it yet | |
| 08:24:50 | sean-k-mooney | ygk_12345: can you post the error message to paste.openstack.org | |
| 08:26:03 | sean-k-mooney | ygk_12345: also on the node with the error can you tell me how much ram do you have, what is the reseved memory set in the config and how much is used by instance on that node? | |
| 08:39:49 | mriedem | dims: http://docs.sqlalchemy.org/en/latest/core/metadata.html?highlight=class%20column#sqlalchemy.schema.Column.params.onupdate | |
| 08:41:12 | mriedem | e.g. Column('last_modified', DateTime, onupdate=func.utc_timestamp()) | |
| 08:58:54 | sean-k-mooney | mdbooth: regarding https://review.openstack.org/#/c/536351 can you confirm that this is never used in the disk filter. looking at https://github.com/openstack/nova/blob/master/nova/scheduler/filters/disk_filter.py#L38-L59 we appear to purely calulate the availe disk from the host_state | |
| 09:01:34 | mdbooth | sean-k-mooney: Don't they have the same source? | |
| 09:02:02 | sean-k-mooney | mdbooth: i can not find any use of the migrate_data.disk_available_mb in the condoctor before the downcall to the compute | |
| 09:03:01 | sean-k-mooney | mdbooth: which the host_state object and where we are constuction the migration data | |
| 09:03:52 | mdbooth | sean-k-mooney: Can you restate your original question. Can I confirm that *what*? is never used... | |
| 09:04:45 | sean-k-mooney | mdbooth: so looking at https://review.openstack.org/#/c/536351/3/nova/virt/libvirt/driver.py the migrate_data object constructed in this fuction is not used before the downcall to the compute node. | |
| 09:05:38 | sean-k-mooney | but there is also a comment about this change relating to the disk filter however the disk filter appears to calulate the availabel disk space independly via the info in the host state object | |
| 09:06:44 | sean-k-mooney | so im trying to figure out at what point is the value store in the migarte_data.disk_available_mb actully used | |
| 09:12:27 | mdbooth | sean-k-mooney: It's used in check_can_live_migrate_source | |
| 09:12:48 | mdbooth | But only if disk_over_commit is set | |
| 09:13:18 | gryf | its verified in _assert_dest_node_has_enough_disk method and only for block_migration | |
| 09:15:51 | sean-k-mooney | mdbooth: where is mdbooth gryf i am not seeing either of those methods in the condoctor live_migrate.py are they performed on the compute node via rpc | |
| 09:16:25 | mdbooth | iirc check_can_live_migrate_source is called from check_can_live_migrate_dest | |
| 09:16:26 | gryf | sean-k-mooney, they are performed in driver | |
| 09:16:39 | mdbooth | Nah, they're called from compute manager | |
| 09:16:46 | gryf | and called from compute manager | |
| 09:16:48 | mdbooth | The code's in the driver | |
| 09:16:49 | gryf | yeah | |
| 09:17:54 | mdbooth | Oh, _dest is called from conductor | |
| 09:18:04 | gryf | compute.manager.check_can_live_migrate_destination | |
| 09:18:16 | mdbooth | Which then calls _source | |
| 09:19:15 | openstackgerrit | Merged openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | |
| 09:20:33 | openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Add placement perf info gathering to end of nova-next https://review.openstack.org/591367 | |
| 09:21:10 | mriedem | mdbooth: https://docs.openstack.org/nova/latest/reference/live-migration.html :) | |
| 09:21:53 | mdbooth | mriedem: It's so simple they even drew a picture about it. | |
| 09:22:52 | sean-k-mooney | mdbooth: gryf so yes looking at https://github.com/openstack/nova/blob/0502b0ffaf20ef3735e633715ba0f9dda976cb7a/nova/virt/libvirt/driver.py#L6774 | |
| 09:23:17 | sean-k-mooney | i do not see the disk allocation ratio being used in that check | |
| 09:24:36 | sean-k-mooney | so would that not imply we have to take account of it when we first calulate disk_avaiable_mb | |
| 09:25:03 | mdbooth | sean-k-mooney: So this code isn't related to DiskFilter | |
| 09:25:07 | gryf | sean-k-mooney, possibly, but that's another thing, isn't it? | |
| 09:25:19 | sean-k-mooney | mdbooth: yes i know | |
| 09:26:02 | sean-k-mooney | gryf: well the bug is fixing how we calulate the available diskspace. if it does so by not allowing over allocation of disk when its turned on that jsut creates another bug | |
| 09:26:33 | gryf | hm. | |
| 09:26:36 | mdbooth | I would hope that disk_available_mb would be calculated as the amount of available disk, in mb. | |
| 09:27:02 | mdbooth | If anybody makes it anything else, I want to know where they live. | |
| 09:27:19 | mdbooth | If we need to take disk allocation ratio into account, we should do that at the point of use. | |
| 09:27:26 | mdbooth | Or call it something else. | |
| 09:27:32 | sean-k-mooney | it should be eitehr available disk = ((total diskspcae - reserved disk space) * disk allocation ratio) - used disk space or available disk = ((total diskspcae - reserved disk space)) - used disk space | |
| 09:28:01 | sean-k-mooney | if its the later then we never take account of disk overcomit when live migrating | |
| 09:28:38 | mdbooth | I'd argue that overcommit has always been a job for the scheduler, anyway. | |
| 09:28:41 | gryf | sean-k-mooney, i'd move out reserved disk space out of equation | |
| 09:28:45 | openstackgerrit | Chris Dent proposed openstack/nova stable/pike: [placement] Retry allocation writes server side https://review.openstack.org/590745 | |
| 09:29:07 | mdbooth | I think live migration should fail only if there aren't physically enough bits on the destination to hold the source bits. | |
| 09:29:08 | gryf | we don;t want to change that by allocation ratio if it's something else from 1 | |
| 09:29:40 | sean-k-mooney | gryf: why not | |
| 09:30:07 | sean-k-mooney | the allocation ratio should only be applied to the portion of the disk that is not reserved | |
| 09:32:15 | gryf | so, isn't it be (total disk space - used disk space - reserved disk space) * allocation ratio? | |
| 09:33:09 | gryf | or even (total disk space - used disk space) * allocation ratio - reserved disk space | |
| 09:33:09 | cdent | "If we need to take disk allocation ratio into account, we should do that at the point of use." yes | |
| 09:33:58 | sean-k-mooney | actully gryf no we dont want to multiple the used disk space by the allocation ratio | |
| 09:34:04 | sean-k-mooney | but its not needed in either case | |
| 09:34:08 | sean-k-mooney | i just realised https://github.com/openstack/nova/blob/0502b0ffaf20ef3735e633715ba0f9dda976cb7a/nova/virt/libvirt/driver.py#L6768 | |
| 09:34:11 | mdbooth | There are 2 ways to get a live migration destination: | |
| 09:34:22 | mdbooth | 1. From the scheduler, which will take this stuff into account | |
| 09:34:27 | sean-k-mooney | is using the actul size of the disk not the virtual size | |
| 09:34:54 | mdbooth | 2. From the admin, who doesn't care about your pesky artificial resource constraints. | |
| 09:35:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consumer gen support for delete instance allocations https://review.openstack.org/591597 | |
| 09:35:53 | sean-k-mooney | in the overcomit case that is so the driver only compares bytes need to copy the disk to actul bytes free on the dest and the schduler/placement operates on teh virtual disk size with allocation ratios taken into account | |
| 09:36:24 | mdbooth | sean-k-mooney: That's the only thing it should ever compare. Anything else is legacy. | |
| 09:37:01 | sean-k-mooney | mdbooth: yes just trying to get the full end to end effect of this change right in my head to make sure there was no edge case missed | |
| 09:37:45 | mdbooth | Right. If an admin wants to move a workload to compute X, the only reason we should say no is if it physically won't fit. Any future problems caused by that are the admin's to deal with. | |
| 09:39:33 | mdbooth | But hopefully the admin only does that if they really, really need to, and usually they ask the scheduler. | |
| 09:39:33 | gryf | mdbooth, by "phisically fit" do you mean current src image size, or the maximum one? | |
| 09:39:46 | mdbooth | gryf: I mean current, allocated disk. | |
| 09:39:50 | gryf | ack | |
| 09:39:57 | sean-k-mooney | mdbooth: gryf cool im much happier this this change then. its sutle but makes sense in context | |
| 09:44:49 | openstackgerrit | Surya Seetharaman proposed openstack/python-novaclient master: [WIP/POC] Client changes for down cell https://review.openstack.org/579563 | |
| 09:54:43 | gryf | sean-k-mooney, do you want me to rebase and correct typo on commit message? | |
| 09:54:49 | gryf | typos* | |
| 09:54:59 | sean-k-mooney | gryf: they are just nits so no its fine | |