| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-25 | |||
| 20:03:11 | melwitt | efried: pack is default | |
| 20:03:28 | efried | looking4somethin: there you go ^ so by default you'll go to one compute until it's "full". | |
| 20:03:43 | efried | If you prefer to spread, you can do things with weighers. | |
| 20:04:01 | melwitt | you need to adjust some config options if you would prefer VMs to spread first onto compute hosts | |
| 20:04:32 | looking4somethin | that's it then. however I have observed some spreading previously but right now one compute is getting everything. | |
| 20:05:52 | melwitt | you can use https://docs.openstack.org/nova/rocky/configuration/config.html#placement.randomize_allocation_candidates to have placement randomize the computes it returns to nova | |
| 20:06:17 | melwitt | you can use https://docs.openstack.org/nova/rocky/configuration/config.html#filter_scheduler.host_subset_size to make nova choose randomly from a larger subset of hosts | |
| 20:06:19 | efried | melwitt: If there's a weigher in play that packs by default, that won't actually help (unless the deployment is bigger than the ?limit) | |
| 20:06:42 | melwitt | oh, I see | |
| 20:06:48 | melwitt | yeah | |
| 20:07:16 | efried | to spread, I thought you had to implement a weigher that weighs emptier hosts heavier. | |
| 20:07:32 | looking4somethin | efried how I can do that | |
| 20:07:42 | melwitt | you may have to adjust individual weighers multipliers if any of them are doing packing behavior | |
| 20:07:45 | efried | blind leading blind here... | |
| 20:07:48 | looking4somethin | to use the emptier hosts above all others | |
| 20:08:10 | looking4somethin | in nova.conf right? | |
| 20:08:13 | melwitt | there is also https://docs.openstack.org/nova/rocky/configuration/config.html#filter_scheduler.shuffle_best_same_weighed_hosts if you want to randomize hosts that are all weighed the same | |
| 20:08:34 | melwitt | yeah nova.conf | |
| 20:08:37 | efried | hm, CPUWeigher has a docstring that says spread is the default. | |
| 20:09:00 | efried | ditto RAMWeigher | |
| 20:09:18 | melwitt | here's the docs for all the weighers so you have to read those to see which one might be causing behavior you don't like by default https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights | |
| 20:09:36 | efried | phew, an rtfm answer, thanks melwitt | |
| 20:10:04 | looking4somethin | thanks melwitt | |
| 20:18:32 | looking4somethin | how can I replace (in nova.conf) nova.scheduler.weights.all_weighers by nova.scheduler.weights.CPUWeigher or just to select one weigher | |
| 20:19:12 | looking4somethin | I want to allocate to a node that have the most vcpus available | |
| 20:24:54 | efried | looking4somethin: It looks to me like you would say | |
| 20:24:55 | efried | weight_classes = 'nova.scheduler.weights.cpu.CPUWeigher' | |
| 20:24:55 | efried | [scheduler] | |
| 20:31:03 | looking4somethin | efried I will try it tomorrow, thanks | |
| 20:31:20 | efried | good luck | |
| 20:31:24 | efried | sweet dreams | |
| 20:31:28 | efried | (are made of this) | |
| 20:34:49 | mriedem | melwitt: efried: isn't that twice this week someone has been in here asking how to randomize allocations? wasn't eandersson asking about the same thing? | |
| 20:35:05 | mriedem | maybe we need a more obvious doc about the various knobs used to make that happen | |
| 20:35:12 | mriedem | since it seems like there are a few things to look at | |
| 20:35:17 | efried | except I think in both cases randomize turned out not to be the answer. | |
| 20:35:37 | mriedem | well, i don't mean the actual placement randomize option, | |
| 20:35:45 | mriedem | but shuffling a host within the same weighted subset | |
| 20:36:06 | eandersson | There are three options, and all seem very similar to me (especially if not reading the source code first) | |
| 20:36:37 | mriedem | my point is, can we get people to (1) stop reading source code and (2) find what they are looking for in the docs? | |
| 20:36:49 | eandersson | 100% this ^ | |
| 20:36:54 | mriedem | i'm thinking about it b/c i've been working on a troubleshooting doc for orphaned allocations | |
| 20:37:11 | mriedem | which i'm about to birth | |
| 20:38:57 | eandersson | btw we did like 6k cold migrations and live migrations with Rocky | |
| 20:39:04 | eandersson | Don't ask me why :p | |
| 20:39:19 | eandersson | And 99% were successful | |
| 20:39:27 | eandersson | We had some really bad failures | |
| 20:39:35 | eandersson | The worst was some sort of neutron port corruption | |
| 20:39:51 | eandersson | where the port was on the wrong compute, and in some cases completely corrupt in the neutron db | |
| 20:40:49 | eandersson | But at least no orphaned placement allocations so that is great | |
| 20:44:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: add troubleshooting guide for cleaning up orphaned allocations https://review.opendev.org/691427 | |
| 20:45:15 | mriedem | eandersson: port issues with the host binding details specifically? because someone else had reported issues with live migrations and weird port binding UniqueConstraint issues | |
| 20:45:34 | eandersson | Yea I think so | |
| 20:45:49 | eandersson | I can probably dig up some more details | |
| 20:45:56 | eandersson | but nothing much in the logs unfortunately | |
| 20:46:58 | eandersson | live migrations actually worked pretty well overall in Rocky | |
| 20:47:26 | eandersson | But only when emptying one host and live migrating all of them into an empty host | |
| 20:47:31 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/691282 | |
| 20:48:10 | mriedem | eandersson: https://review.opendev.org/#/c/653506/ | |
| 20:48:26 | mriedem | ^ was the solution for the other person that reported that issue | |
| 20:49:05 | mriedem | so far obviously there hasn't been enough motivation to pick that back up and address comments | |
| 20:50:07 | eandersson | I am not too worried about it at the moment as live migrations only works in the above scenario for us | |
| 20:50:23 | eandersson | but maybe I can upgrade to Train in the lab and see if I can reproduce it. | |
| 20:50:32 | eandersson | And test that patch out. | |
| 21:08:00 | mriedem | i'm out | |
| 21:08:01 | mriedem | o/ | |
| 23:37:38 | openstackgerrit | Merged openstack/nova stable/rocky: cleanup evacuated instances not on hypervisor https://review.opendev.org/687550 | |
| 23:37:44 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional test for resize crash compute restart revert https://review.opendev.org/687563 | |
| 23:37:52 | openstackgerrit | Merged openstack/nova stable/rocky: Prevent init_host test to interfere with other tests https://review.opendev.org/687862 | |
| #openstack-nova - 2019-10-26 | |||
| 00:08:25 | eandersson | > ('Connection aborted.', BadStatusLine("''",)) | |
| 00:08:44 | eandersson | I have seen a few bugs mentioning keepalive causing this issue (e.g. when computes are talking to neutron) | |
| 00:09:15 | eandersson | Anyone confirmed that disabling keepalive fixes it? | |
| 00:19:41 | openstack | bug 1833581 in OpenStack Compute (nova) rocky "instance stuck in BUILD state if nova-compute is restarted" [Low,In progress] https://launchpad.net/bugs/1833581 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 00:19:41 | openstackgerrit | Merged openstack/nova stable/rocky: Functional reproduce for bug 1833581 https://review.opendev.org/687564 | |
| 00:19:45 | openstack | bug 1849409 in OpenStack Compute (nova) stein "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | |
| 00:19:45 | openstackgerrit | Merged openstack/nova stable/stein: Add functional regression test for bug 1849409 https://review.opendev.org/690715 | |
| 00:50:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | |
| 01:45:53 | openstackgerrit | Merged openstack/nova stable/stein: Fix listing deleted servers with a marker https://review.opendev.org/690716 | |
| 01:48:49 | openstackgerrit | kevinzhao proposed openstack/nova master: Nova: node should be deleted when nova-compute is removed https://review.opendev.org/671731 | |
| 01:57:53 | openstackgerrit | Merged openstack/nova master: Remove redundant call to get/create default security group https://review.opendev.org/688206 | |
| 04:53:46 | openstackgerrit | Merged openstack/nova stable/rocky: Error out interrupted builds https://review.opendev.org/687565 | |
| 05:35:09 | openstackgerrit | Merged openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | |
| 05:55:42 | openstackgerrit | Merged openstack/python-novaclient master: Stop supporting and testing python2 https://review.opendev.org/691365 | |
| 06:00:47 | mordred | efried: ^^ that's such an exciting patch | |
| #openstack-nova - 2019-10-27 | |||
| 08:45:01 | openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.opendev.org/526900 | |
| 10:00:41 | openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.opendev.org/526900 | |
| 13:08:10 | openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.opendev.org/526900 | |
| #openstack-nova - 2019-10-28 | |||
| 01:26:31 | openstackgerrit | Merged openstack/os-traits master: Stop testing with py2 and switch to ussuri jobs https://review.opendev.org/690997 | |
| 01:28:07 | openstackgerrit | Merged openstack/os-resource-classes master: Stop testing with py2 https://review.opendev.org/690996 | |
| 08:32:21 | bauzas | good morning Nov | |
| 08:32:23 | bauzas | Nova* | |
| 08:37:58 | openstackgerrit | sunjiyun-inspur proposed openstack/nova master: On branch vm-query-notsameproject https://review.opendev.org/691629 | |
| 09:35:55 | openstackgerrit | Kobi Samoray proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247 | |
| 09:50:31 | dpawlik | Good morning. If you have some free time, pls check https://review.opendev.org/#/c/690482/ . Thanks! | |
| 10:07:36 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: cleanup evacuated instances not on hypervisor https://review.opendev.org/687873 | |
| 10:08:31 | openstackgerrit | Silvan Kaiser proposed openstack/nova master: [WIP] Move Nova Quobyte driver to LibvirtMountedFileSystemVolumeDriver https://review.opendev.org/687066 | |
| 10:10:31 | openstackgerrit | Qiu Fossen proposed openstack/nova-specs master: support instance tag fuzzy query https://review.opendev.org/691651 | |
| 10:13:59 | kaisers | mdbooth: I've a change in flight to move the Quobyte driver to LibvirtMountedFileSystemVolumeDriver (https://review.opendev.org/#/c/687066/). It's still wip but I't be great if you could have a look if the approach for the systemd related mount in there looks sane to you. | |
| 10:35:56 | mdbooth | kaisers: Sure. Will try to look later today. | |
| 11:15:59 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server https://review.opendev.org/580336 | |