| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-25 | |||
| 19:42:03 | mriedem | we SEV1 IPL'ed that APAR | |
| 19:43:10 | mriedem | efried: those py2 removal patches are going to be the end of one of us | |
| 19:43:48 | efried | mriedem: I figured there would be some churn. I'm a zuul dunce to begin with. | |
| 19:44:11 | efried | but I feel like we're ahead of the game anyway, we've got months to get it sorted. | |
| 19:44:42 | mriedem | way to jinx yourself | |
| 19:47:40 | openstackgerrit | Eric Fried proposed openstack/os-vif master: Drop python2 support and testing https://review.opendev.org/691364 | |
| 19:51:02 | melwitt | IPL and APAR, had to google those | |
| 19:57:58 | efried | mriedem: join us in -release if you please? | |
| 19:58:42 | looking4somethin | yes | |
| 19:59:01 | looking4somethin | mriedem are you talking to me? | |
| 20:01:00 | efried | looking4somethin: sorry, you arrived in the middle of a conversation | |
| 20:01:26 | looking4somethin | awesome! | |
| 20:02:04 | efried | looking4somethin: Have you tried randomizing your placement allocation candidates? | |
| 20:02:23 | efried | landing on one compute isn't really a bug | |
| 20:02:40 | efried | I can't remember if we pack or spread by default -- dansmith? | |
| 20:02:47 | looking4somethin | not really | |
| 20:02:51 | looking4somethin | how can I do it. | |
| 20:03:11 | melwitt | efried: pack is default | |
| 20:03:28 | efried | looking4somethin: there you go ^ so by default you'll go to one compute until it's "full". | |
| 20:03:43 | efried | If you prefer to spread, you can do things with weighers. | |
| 20:04:01 | melwitt | you need to adjust some config options if you would prefer VMs to spread first onto compute hosts | |
| 20:04:32 | looking4somethin | that's it then. however I have observed some spreading previously but right now one compute is getting everything. | |
| 20:05:52 | melwitt | you can use https://docs.openstack.org/nova/rocky/configuration/config.html#placement.randomize_allocation_candidates to have placement randomize the computes it returns to nova | |
| 20:06:17 | melwitt | you can use https://docs.openstack.org/nova/rocky/configuration/config.html#filter_scheduler.host_subset_size to make nova choose randomly from a larger subset of hosts | |
| 20:06:19 | efried | melwitt: If there's a weigher in play that packs by default, that won't actually help (unless the deployment is bigger than the ?limit) | |
| 20:06:42 | melwitt | oh, I see | |
| 20:06:48 | melwitt | yeah | |
| 20:07:16 | efried | to spread, I thought you had to implement a weigher that weighs emptier hosts heavier. | |
| 20:07:32 | looking4somethin | efried how I can do that | |
| 20:07:42 | melwitt | you may have to adjust individual weighers multipliers if any of them are doing packing behavior | |
| 20:07:45 | efried | blind leading blind here... | |
| 20:07:48 | looking4somethin | to use the emptier hosts above all others | |
| 20:08:10 | looking4somethin | in nova.conf right? | |
| 20:08:13 | melwitt | there is also https://docs.openstack.org/nova/rocky/configuration/config.html#filter_scheduler.shuffle_best_same_weighed_hosts if you want to randomize hosts that are all weighed the same | |
| 20:08:34 | melwitt | yeah nova.conf | |
| 20:08:37 | efried | hm, CPUWeigher has a docstring that says spread is the default. | |
| 20:09:00 | efried | ditto RAMWeigher | |
| 20:09:18 | melwitt | here's the docs for all the weighers so you have to read those to see which one might be causing behavior you don't like by default https://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights | |
| 20:09:36 | efried | phew, an rtfm answer, thanks melwitt | |
| 20:10:04 | looking4somethin | thanks melwitt | |
| 20:18:32 | looking4somethin | how can I replace (in nova.conf) nova.scheduler.weights.all_weighers by nova.scheduler.weights.CPUWeigher or just to select one weigher | |
| 20:19:12 | looking4somethin | I want to allocate to a node that have the most vcpus available | |
| 20:24:54 | efried | looking4somethin: It looks to me like you would say | |
| 20:24:55 | efried | weight_classes = 'nova.scheduler.weights.cpu.CPUWeigher' | |
| 20:24:55 | efried | [scheduler] | |
| 20:31:03 | looking4somethin | efried I will try it tomorrow, thanks | |
| 20:31:20 | efried | good luck | |
| 20:31:24 | efried | sweet dreams | |
| 20:31:28 | efried | (are made of this) | |
| 20:34:49 | mriedem | melwitt: efried: isn't that twice this week someone has been in here asking how to randomize allocations? wasn't eandersson asking about the same thing? | |
| 20:35:05 | mriedem | maybe we need a more obvious doc about the various knobs used to make that happen | |
| 20:35:12 | mriedem | since it seems like there are a few things to look at | |
| 20:35:17 | efried | except I think in both cases randomize turned out not to be the answer. | |
| 20:35:37 | mriedem | well, i don't mean the actual placement randomize option, | |
| 20:35:45 | mriedem | but shuffling a host within the same weighted subset | |
| 20:36:06 | eandersson | There are three options, and all seem very similar to me (especially if not reading the source code first) | |
| 20:36:37 | mriedem | my point is, can we get people to (1) stop reading source code and (2) find what they are looking for in the docs? | |
| 20:36:49 | eandersson | 100% this ^ | |
| 20:36:54 | mriedem | i'm thinking about it b/c i've been working on a troubleshooting doc for orphaned allocations | |
| 20:37:11 | mriedem | which i'm about to birth | |
| 20:38:57 | eandersson | btw we did like 6k cold migrations and live migrations with Rocky | |
| 20:39:04 | eandersson | Don't ask me why :p | |
| 20:39:19 | eandersson | And 99% were successful | |
| 20:39:27 | eandersson | We had some really bad failures | |
| 20:39:35 | eandersson | The worst was some sort of neutron port corruption | |
| 20:39:51 | eandersson | where the port was on the wrong compute, and in some cases completely corrupt in the neutron db | |
| 20:40:49 | eandersson | But at least no orphaned placement allocations so that is great | |
| 20:44:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: add troubleshooting guide for cleaning up orphaned allocations https://review.opendev.org/691427 | |
| 20:45:15 | mriedem | eandersson: port issues with the host binding details specifically? because someone else had reported issues with live migrations and weird port binding UniqueConstraint issues | |
| 20:45:34 | eandersson | Yea I think so | |
| 20:45:49 | eandersson | I can probably dig up some more details | |
| 20:45:56 | eandersson | but nothing much in the logs unfortunately | |
| 20:46:58 | eandersson | live migrations actually worked pretty well overall in Rocky | |
| 20:47:26 | eandersson | But only when emptying one host and live migrating all of them into an empty host | |
| 20:47:31 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: libvirt: Ignore volume exceptions during post_live_migration https://review.opendev.org/691282 | |
| 20:48:10 | mriedem | eandersson: https://review.opendev.org/#/c/653506/ | |
| 20:48:26 | mriedem | ^ was the solution for the other person that reported that issue | |
| 20:49:05 | mriedem | so far obviously there hasn't been enough motivation to pick that back up and address comments | |
| 20:50:07 | eandersson | I am not too worried about it at the moment as live migrations only works in the above scenario for us | |
| 20:50:23 | eandersson | but maybe I can upgrade to Train in the lab and see if I can reproduce it. | |
| 20:50:32 | eandersson | And test that patch out. | |
| 21:08:00 | mriedem | i'm out | |
| 21:08:01 | mriedem | o/ | |
| 23:37:38 | openstackgerrit | Merged openstack/nova stable/rocky: cleanup evacuated instances not on hypervisor https://review.opendev.org/687550 | |
| 23:37:44 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional test for resize crash compute restart revert https://review.opendev.org/687563 | |
| 23:37:52 | openstackgerrit | Merged openstack/nova stable/rocky: Prevent init_host test to interfere with other tests https://review.opendev.org/687862 | |
| #openstack-nova - 2019-10-26 | |||
| 00:08:25 | eandersson | > ('Connection aborted.', BadStatusLine("''",)) | |
| 00:08:44 | eandersson | I have seen a few bugs mentioning keepalive causing this issue (e.g. when computes are talking to neutron) | |
| 00:09:15 | eandersson | Anyone confirmed that disabling keepalive fixes it? | |
| 00:19:41 | openstack | bug 1833581 in OpenStack Compute (nova) rocky "instance stuck in BUILD state if nova-compute is restarted" [Low,In progress] https://launchpad.net/bugs/1833581 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 00:19:41 | openstackgerrit | Merged openstack/nova stable/rocky: Functional reproduce for bug 1833581 https://review.opendev.org/687564 | |
| 00:19:45 | openstack | bug 1849409 in OpenStack Compute (nova) stein "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | |
| 00:19:45 | openstackgerrit | Merged openstack/nova stable/stein: Add functional regression test for bug 1849409 https://review.opendev.org/690715 | |
| 00:50:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | |
| 01:45:53 | openstackgerrit | Merged openstack/nova stable/stein: Fix listing deleted servers with a marker https://review.opendev.org/690716 | |
| 01:48:49 | openstackgerrit | kevinzhao proposed openstack/nova master: Nova: node should be deleted when nova-compute is removed https://review.opendev.org/671731 | |
| 01:57:53 | openstackgerrit | Merged openstack/nova master: Remove redundant call to get/create default security group https://review.opendev.org/688206 | |
| 04:53:46 | openstackgerrit | Merged openstack/nova stable/rocky: Error out interrupted builds https://review.opendev.org/687565 | |
| 05:35:09 | openstackgerrit | Merged openstack/nova master: Fix policy doc for host_status and extended servers attribute https://review.opendev.org/689833 | |
| 05:55:42 | openstackgerrit | Merged openstack/python-novaclient master: Stop supporting and testing python2 https://review.opendev.org/691365 | |