| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 21:44:30 | mriedem | is that instance in the db? | |
| 21:44:52 | mnaser | hmm | |
| 21:45:02 | mriedem | the code calls out 2 cases that could happen: | |
| 21:45:03 | mnaser | no, i think the exception i was seeing was unrelated | |
| 21:45:08 | mnaser | i think it was no placement + vm deleted | |
| 21:45:23 | mnaser | i think i jumped to a conclusion there | |
| 21:45:25 | mriedem | 1. compute RT is racing with the scheduler where the scheduler created allocations in placement but didn't yet create the instance in the cell db, | |
| 21:45:45 | mriedem | 2. the instance was deleted and archived/purged from the db, but the alloctions are still in placement for that node | |
| 21:45:52 | mriedem | yeah, | |
| 21:46:03 | mriedem | so if the compute couldn't talk to placement when the vm was deleted, we'd fail to cleanup the allocations | |
| 21:46:32 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete | |
| 21:46:44 | mnaser | thanks for making life easy and writing those clie stuf | |
| 21:46:46 | mriedem | openstack resource provider allocation delete 13a09ff5-7124-4b3c-8229-a8691a2f43b4 | |
| 21:46:57 | mriedem | if the instance is truly deleted | |
| 21:47:13 | mriedem | if it were moved and allocations were just messed up, doing ^ would be bad | |
| 21:47:42 | mnaser | ill double check they're all deleted | |
| 21:47:45 | mriedem | might be good to update that log message with the same info | |
| 21:47:55 | mnaser | tail -1000 /var/log/nova/nova-compute.log | grep 'has allocations against this compute host but is not found in the database.' | cut -d' ' -f13 | sort | uniq | xargs -n1 echo nova show | |
| 21:47:56 | mnaser | magic | |
| 21:48:30 | mriedem | "If you continue to see this message and have confirmed the instance is truly gone from the database, you can run 'openstack resource provider allocation delete <uuid>' to remove the allocation from the placement service." | |
| 21:48:46 | mnaser | yeah just even document it somewhere i guess | |
| 21:48:54 | mnaser | i mean | |
| 21:48:56 | mriedem | well, | |
| 21:48:58 | mnaser | couldnt nova confirm that it doesnt exist in the db | |
| 21:49:05 | mriedem | not if it's a race | |
| 21:49:06 | mnaser | and technically just delete it when it polls | |
| 21:49:09 | mnaser | ah | |
| 21:49:21 | mnaser | i guess we'd have to check N times that it's not there but that starts becoming silly i guess | |
| 21:49:26 | mriedem | as noted, we create the allocations in scheduling before the instance is created in the cell db | |
| 21:49:33 | mriedem | right | |
| 21:49:47 | mnaser | or if its been running for a while | |
| 21:49:53 | mriedem | putting it into some kind of faq could work, but people might not see that and come here asking the same thing | |
| 21:49:59 | mnaser | like 300s age | |
| 21:50:08 | mnaser | 300s isn't a race condition at that point i guess | |
| 21:50:24 | mnaser | but i guess log is the easiest path | |
| 21:50:31 | mriedem | rather than nova auto-delete your stuff, i'd prefer to just document weird edge cases for manual intervention | |
| 21:51:07 | mriedem | we've also talked about (last week it came up too) a command to compare placement allocations against nova instances, | |
| 21:51:13 | mriedem | the opposite of heal_allocations, | |
| 21:51:24 | mriedem | more of a "tell me what garbage is in the placement db b/c nova didn't clean up after itself" | |
| 21:51:54 | mriedem | but we can't reliably do that until we have consumer types in placement... | |
| 21:54:47 | mriedem | takashin: is this todo still valid https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3509 given we check the specified host is in the same cell above? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3405 | |
| 21:55:55 | mriedem | i think we can remove that todo; when we call objects.ComputeNode.get_first_node_by_host_for_old_compat the context is targeted to the cell in which the instance currently exists | |
| 21:56:08 | mriedem | so if the specified host is in another cell, we'll get ComputeHostNotFound there | |
| 21:58:57 | mriedem | johnthetubaguy: we should remove this rebuild parameter from the conductor migrate_server task API :) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L263 | |
| 21:59:12 | mriedem | that must have been something someone talked about way back in 2013 to unify the various move APIs | |
| 22:10:08 | takashin | mriedem: you are right. We can remove the 'todo'. | |
| 22:10:42 | mriedem | takashin: please push a change to remove it and i'll +2 | |
| 22:11:11 | takashin | mriedem: okay. I will submit the patch. | |
| 22:13:52 | vishwanathj | hi looking for guidance on how to limit the number of instances on certain set of hosts... | |
| 22:15:06 | vishwanathj | not clear on sequence of steps to be executed on controller and compute nodes to get this working | |
| 22:31:47 | mriedem | there is a NumInstancesFilter | |
| 22:32:11 | mriedem | vishwanathj: https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#numinstancesfilter | |
| 22:32:25 | mriedem | and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregatenuminstancesfilter for aggregates | |
| 22:33:16 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | |
| 22:33:53 | openstack | Launchpad bug 1740320 in OpenStack Compute (nova) "nova-scheduler does not honor max_instances_per_host set to a host aggregate" [Undecided,Confirmed] | |
| 22:33:53 | mriedem | there is a known bug with those filters https://bugs.launchpad.net/nova/+bug/1740320 | |
| 22:34:01 | mriedem | race bug | |
| 22:47:40 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove an unnecessary comment https://review.openstack.org/603924 | |
| 22:49:34 | takashin | mriedem: I submitted https://review.openstack.org/#/c/603924 . | |
| 22:57:15 | sorrison | Anyone know how to re kick a gate job that failed due to a timeout. https://review.openstack.org/#/c/526558/ I tried a recheck but that aint it | |
| 22:57:42 | sorrison | mriedem: ^ | |
| 23:02:34 | mriedem | sorrison: the gate has been eating a ton of dairy | |
| 23:02:49 | mriedem | plug 526558 into http://zuul.openstack.org/ | |
| 23:03:06 | mriedem | it's queued up | |
| 23:03:54 | mriedem | http://lists.openstack.org/pipermail/openstack-dev/2018-September/134867.html | |
| 23:04:48 | sorrison | ok thanks | |
| 23:19:24 | mriedem | takashin: got it, thanks | |
| 23:20:11 | takashin | mriedem: Thank you for your review. | |
| 23:40:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930 | |
| 23:40:39 | mriedem | ship it! | |
| 23:52:34 | gmann | lbragstad: i will check the ML today and get back to you. | |
| #openstack-nova - 2018-09-20 | |||
| 00:57:24 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | |
| 01:07:32 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Spec: allocation candidates in tree https://review.openstack.org/603585 | |
| 01:12:57 | openstackgerrit | Tao Li proposed openstack/python-novaclient master: Remove the unused instance-name https://review.openstack.org/602520 | |
| 01:35:04 | lbragstad | gmann sounds good | |
| 02:33:47 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955 | |
| 02:50:24 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | |
| 02:57:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | |
| 03:12:07 | openstackgerrit | fupingxie proposed openstack/python-novaclient master: Do not translate log messages. https://review.openstack.org/603962 | |
| 04:16:51 | openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API inconsistency cleanup https://review.openstack.org/603969 | |
| 04:28:21 | openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API inconsistency cleanup https://review.openstack.org/603969 | |
| 04:47:10 | gmann | lyarwood: i replied on this - https://review.openstack.org/#/c/602018/ | |
| 04:47:30 | gmann | lyarwood: stable branch patches needs to get in before openstack-infra/openstack-zuul-jobs one | |
| 06:28:44 | openstackgerrit | huanhongda proposed openstack/nova stable/ocata: Cleanup RP and HM records while deleting a compute service. https://review.openstack.org/603749 | |
| 06:29:14 | openstackgerrit | fupingxie proposed openstack/nova master: Avoid Forcing the Translation of Translatable Variables https://review.openstack.org/603993 | |
| 06:47:48 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Only update cells' transport_url and database_connection when needed. https://review.openstack.org/603998 | |
| 06:53:45 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add support for Windows network commands https://review.openstack.org/487405 | |
| 07:38:16 | openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | |
| 07:44:23 | bauzas | good morning | |
| 07:45:33 | sean-k-mooney | good morning :) | |
| 07:46:10 | gibi | good morning folks | |
| 08:26:23 | openstackgerrit | Merged openstack/nova master: libvirt: Drop MIN_LIBVIRT_VHOSTUSER_MQ https://review.openstack.org/565472 | |
| 08:49:15 | kashyap | Hey folks, I wonder if anyone here knows Linux systems where only /dev/random exists | |
| 08:49:19 | kashyap | And not /dev/urandom. | |
| 08:50:20 | kashyap | Because I'm considering to add a default 'virtio-rng' device for Nova guests, whic uses /dev/urandom. | |
| 08:50:24 | openstack | Launchpad bug 1789868 in OpenStack Compute (nova) "RFE: Add a virtio-rng device to Nova instances by default" [Medium,Triaged] - Assigned to Kashyap Chamarthy (kashyapc) | |
| 08:50:24 | kashyap | https://bugs.launchpad.net/nova/+bug/1789868 | |
| 08:52:03 | kashyap | (If anyone is wondering why '/dev/urandom'? It is non-blocking, and doesn't have the same limitations of `/dev/random`, which is a legacy interface. | |
| 09:02:20 | openstackgerrit | huanhongda proposed openstack/nova stable/ocata: Cleanup the caches when deleting a resource provider https://review.openstack.org/604021 | |
| 09:13:35 | kashyap | stephenfin: The last clean up (IIRC) in that series: https://review.openstack.org/#/c/562313/ (libvirt: Drop MIN_LIBVIRT_PARALLELS_SET_ADMIN_PASSWD) | |