| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-17 | |||
| 22:49:20 | melwitt | but the duplicate, I've only seen that when one nova-manage is racing against another. do you have the discover_hosts scheduler periodic enabled? | |
| 22:49:44 | mnaser | I think we enable that based on something | |
| 22:49:45 | mnaser | one sec | |
| 22:49:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.openstack.org/643664 | |
| 22:50:08 | melwitt | that's another possibility, if you were unlucky enough to have that fire at the same time as another scheduler, or if you were to run 'nova-manage cellv2 discover_hosts' at the same time a scheduler periodic discover_hosts fires | |
| 22:50:39 | mnaser | scheduler.discover_hosts_in_cells_interval = 60 | |
| 22:50:55 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899 | |
| 22:51:07 | mnaser | does discover_hosts run in the conductor I assume? | |
| 22:51:07 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899 | |
| 22:51:20 | melwitt | ok, and is that set on all schedulers or just one? (recommendation is to enable it on only one) see https://review.openstack.org/651947 | |
| 22:51:25 | melwitt | scheduler | |
| 22:52:25 | mnaser | http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-scheduler.log.txt.gz#_2019-04-17_21_52_03_776 | |
| 22:52:30 | mnaser | yeah its around the same time | |
| 22:52:35 | mnaser | we have it enabled on all hosts :X | |
| 22:53:13 | melwitt | ok. yeah, I don't think we knew this was going to be a potential problem, or at least I didn't know | |
| 22:53:22 | mnaser | I mean its the first time I see this | |
| 22:53:24 | mnaser | but at least we kno wwhy | |
| 22:53:26 | melwitt | the trace isn't harmful, it just means discover_hosts collided with another one | |
| 22:53:43 | melwitt | so the patch I linked earlier aims to warn and give guidance instead of tracing | |
| 22:54:15 | mnaser | yeah that would be good because the ansible obviously fails when it sees that | |
| 22:54:18 | mnaser | so that would be good to know | |
| 22:54:24 | melwitt | commit message explains why to do that too vs ignore DBDuplicateEntry | |
| 22:54:30 | mnaser | neat | |
| 22:54:56 | melwitt | oh it does? how do yall do discover_hosts? keep running until you receive status code 0? that's what tripleo does | |
| 22:55:39 | melwitt | the status code is a little weird on that command where non-zero means "there's more work to do" and zero means "I'm done" | |
| 22:55:54 | melwitt | and I think that's the review feedback I got on that patch currently is to enhance the doc to explain that | |
| 22:57:13 | openstackgerrit | Merged openstack/nova master: Stop handling cells v1 in '/os-servers' API https://review.openstack.org/651293 | |
| 22:57:21 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional recreate test for bug 1819963 https://review.openstack.org/648409 | |
| 22:57:22 | openstack | bug 1819963 in OpenStack Compute (nova) rocky "Reverting a resize does not update the instance.availability_zone value to the source az" [Medium,In progress] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem) | |
| 22:57:36 | openstackgerrit | Merged openstack/nova stable/rocky: Update instance.availability_zone on revertResize https://review.openstack.org/648410 | |
| 22:57:48 | mnaser | melwitt: we run discover_hosts every time once only on one of the conductors | |
| 22:58:03 | mnaser | I didn't know discover_hosts does work in batches | |
| 22:58:06 | melwitt | oh, so always expect to receive 0 | |
| 22:58:29 | melwitt | I don't think it does. tripleo retries everything | |
| 22:58:43 | mnaser | https://github.com/openstack/openstack-ansible-os_nova/blob/d1aa5839b482c7412404ebb7bf050ee806fbf281/tasks/nova_db_post_setup.yml#L21-L26 | |
| 22:59:06 | mnaser | ya I try to avoid, uh, brute forcing stuff :p | |
| 22:59:15 | melwitt | yeah it doesn't batch. it's just that non-zero means "I couldn't do the work I needed to do" | |
| 23:00:06 | melwitt | ok... so this will be a problem. we had been thinking to still return 1 if a collision happens i.e. not change the behavior | |
| 23:00:31 | mnaser | right, we can adapt our tasks easily then | |
| 23:00:40 | melwitt | but if you happen to run it while your scheduler does a discover_hosts, you will get a fail status code | |
| 23:00:54 | mnaser | yeah, our case is not parallel but racing with scheduler | |
| 23:01:02 | melwitt | yeah | |
| 23:01:42 | melwitt | ok, well if you want/need different behavior than is currently proposed, please add a comment on the patch review | |
| 23:02:01 | mnaser | ill add comments that we hit this | |
| 23:02:18 | melwitt | example: specific non-zero code that means "collision detected" | |
| 23:02:25 | melwitt | ok | |
| 23:05:55 | mnaser | melwitt: let a comment, danke :) | |
| 23:06:42 | melwitt | kewl | |
| #openstack-nova - 2019-04-18 | |||
| 01:37:18 | imacdonn | anyone know how to trigger GMR when nova(-api) is running under uWSGI? Seems that uWSGI has its own interpretation of SIGUSR2 :/ | |
| 03:26:25 | openstackgerrit | ya.wang proposed openstack/nova-specs master: Expose auto converge and post copy https://review.openstack.org/651681 | |
| 05:12:50 | openstackgerrit | Merged openstack/nova master: Remove 'nova-manage cell' commands https://review.openstack.org/651294 | |
| 05:12:57 | openstackgerrit | Merged openstack/nova master: Stop handling cells v1 for console authentication https://review.openstack.org/651295 | |
| 05:45:48 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.openstack.org/631243 | |
| 05:45:48 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242 | |
| 05:45:49 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.openstack.org/631245 | |
| 05:45:49 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.openstack.org/631244 | |
| 05:58:48 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.openstack.org/651969 | |
| 06:04:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.openstack.org/637716 | |
| 06:38:28 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242 | |
| 06:38:29 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.openstack.org/631244 | |
| 06:38:29 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.openstack.org/631243 | |
| 06:38:30 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.openstack.org/631245 | |
| 07:31:58 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645520 | |
| 08:01:28 | openstackgerrit | Hamdy Khader proposed openstack/nova master: Do not perform port update in case of baremetal instance. https://review.openstack.org/649345 | |
| 08:15:39 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Fix a description for config_drive parameter https://review.openstack.org/653683 | |
| 09:04:47 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Plumbing for locking an instance with reason https://review.openstack.org/653691 | |
| 09:04:47 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: [WIP] Support adding the reason behind a server lock https://review.openstack.org/648662 | |
| 09:14:47 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.openstack.org/649963 | |
| 09:16:59 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.openstack.org/651969 | |
| 09:19:24 | plestang | Hi all! | |
| 09:19:49 | plestang | I have a question regarding nova-conductor | |
| 09:20:29 | plestang | regurlarly compute hosts are requesting for unconfirmed migration by dest compute | |
| 09:21:07 | plestang | The function called is here: https://github.com/openstack/nova/blob/03322bb517925a9f5a04ebdb41c3fd31e7962440/nova/db/sqlalchemy/api.py#L4313 | |
| 09:21:23 | plestang | the read_deleted params is set to yes | |
| 09:21:30 | plestang | in model_query | |
| 09:21:50 | plestang | it generates some load on our infra | |
| 09:22:08 | plestang | because we have around 300000 lines in that table | |
| 09:22:59 | plestang | so all compute hosts of our infra reads regularly all the database | |
| 09:24:45 | plestang | when I explain the request | |
| 09:25:16 | plestang | +------+---------------+------------+--------+-----------------+--------+-----------+--------+--------+-------------+ | |
| 09:25:18 | plestang | | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | | |
| 09:25:20 | plestang | |------+---------------+------------+--------+-----------------+--------+-----------+--------+--------+-------------| | |
| 09:25:22 | plestang | | 1 | SIMPLE | migrations | ALL | <null> | <null> | <null> | <null> | 261177 | Using where | | |
| 09:25:24 | plestang | +------+---------------+------------+--------+-----------------+--------+-----------+--------+--------+-------------+ | |
| 09:25:57 | plestang | I was asking if it were a good idea to set the read_deleted to no instead of yes | |
| 09:26:40 | plestang | it would make the MariaDB query plan use one of the index that use the delete column | |
| 09:27:52 | plestang | if I add "and deleted=0" the query plan becomes | |
| 09:27:54 | plestang | +------+---------------+------------+--------+---------------------------------------------------------------------------------+-----------------------------------------+-----------+-------+--------+------------------------------------+ | |
| 09:27:57 | plestang | | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | | |
| 09:27:59 | plestang | |------+---------------+------------+--------+---------------------------------------------------------------------------------+-----------------------------------------+-----------+-------+--------+------------------------------------| | |
| 09:28:01 | plestang | | 1 | SIMPLE | migrations | ref | migrations_by_host_nodes_and_status_idx,migrations_instance_uuid_and_status_idx | migrations_instance_uuid_and_status_idx | 5 | const | 23610 | Using index condition; Using where | | |
| 09:28:03 | plestang | +------+---------------+------------+--------+---------------------------------------------------------------------------------+-----------------------------------------+-----------+-------+--------+------------------------------------+ | |
| 09:28:54 | plestang | in other word the question is what is the idea behind requesting finished migration in deleted status for compute host | |
| 10:35:00 | aspiers | plestang: you might have more luck when the Americans come online | |
| 10:35:15 | aspiers | kashyap: any thoughts on how to split the getDomainCapabilities() patch up? | |
| 10:35:34 | kashyap | aspiers: Hiya, I want to spend some careful time on it | |
| 10:35:58 | kashyap | aspiers: Right now, I'm chugging through a few things that I "need" to finish | |
| 10:36:03 | aspiers | I guess I could manufacture something to test which isn't ever used in the real world | |
| 10:36:07 | kashyap | aspiers: Rest assured, since I have a "vested interest" in it | |
| 10:36:08 | aspiers | OK np | |