Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-17
22:33:42 melwitt are you running discover_hosts in parallel during a deployment?
22:34:59 melwitt https://bugs.launchpad.net/nova/+bug/1824445
22:35:00 openstack Launchpad bug 1824445 in OpenStack Compute (nova) "nova-manage cellv2 discover_hosts traces when run in parallel" [Low,In progress] - Assigned to melanie witt (melwitt)
22:36:34 mnaser melwitt: nope, should be once only
22:36:55 mnaser but also is it normal that there is two log lines that say compute node created
22:37:58 mnaser melwitt: we also delegate_to the nova-conductor host so we run it once
22:39:05 melwitt I've only seen this before when the command is run in parallel presumably on different hosts
22:39:40 melwitt that's not normal, it shouldn't be trying to create the host mapping for the same host twice
22:39:52 melwitt I don't even get how that could happen non-parallel
22:39:58 mnaser yeah..
22:40:05 mnaser and the pid is the same too
22:41:17 melwitt I dunno, seems like dark magic
22:42:09 melwitt hm, ok the last log is ringing a bell
22:42:36 melwitt did you delete a service via the API and then not restart the nova-compute that goes with it?
22:43:16 melwitt because nova-compute will only create a compute node record when it starts up, it will not recover if the compute node record gets deleted out from under it while it's running
22:46:19 mnaser we don't delete anything afaik :\
22:46:27 mnaser during the CI run at least
22:46:51 melwitt ok. I don't see how that would be related to the discover_hosts thing anyway if it were happening
22:48:29 mnaser https://github.com/openstack/openstack-ansible-os_nova/blob/d1aa5839b482c7412404ebb7bf050ee806fbf281/tasks/main.yml#L233-L243
22:48:42 melwitt mnaser: oh, sorry, I see now it's saying Checking first and then Creating, so that's probably normal
22:48:51 mnaser oh oops
22:48:54 melwitt I thought it was the same message twice but it's not
22:49:10 mnaser you're right yes
22:49:12 mnaser "Compute node record created" is just once
22:49:20 melwitt but the duplicate, I've only seen that when one nova-manage is racing against another. do you have the discover_hosts scheduler periodic enabled?
22:49:44 mnaser I think we enable that based on something
22:49:45 mnaser one sec
22:49:48 openstackgerrit Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.openstack.org/643664
22:50:08 melwitt that's another possibility, if you were unlucky enough to have that fire at the same time as another scheduler, or if you were to run 'nova-manage cellv2 discover_hosts' at the same time a scheduler periodic discover_hosts fires
22:50:39 mnaser scheduler.discover_hosts_in_cells_interval = 60
22:50:55 openstackgerrit Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899
22:51:07 openstackgerrit Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899
22:51:07 mnaser does discover_hosts run in the conductor I assume?
22:51:20 melwitt ok, and is that set on all schedulers or just one? (recommendation is to enable it on only one) see https://review.openstack.org/651947
22:51:25 melwitt scheduler
22:52:25 mnaser http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-scheduler.log.txt.gz#_2019-04-17_21_52_03_776
22:52:30 mnaser yeah its around the same time
22:52:35 mnaser we have it enabled on all hosts :X
22:53:13 melwitt ok. yeah, I don't think we knew this was going to be a potential problem, or at least I didn't know
22:53:22 mnaser I mean its the first time I see this
22:53:24 mnaser but at least we kno wwhy
22:53:26 melwitt the trace isn't harmful, it just means discover_hosts collided with another one
22:53:43 melwitt so the patch I linked earlier aims to warn and give guidance instead of tracing
22:54:15 mnaser yeah that would be good because the ansible obviously fails when it sees that
22:54:18 mnaser so that would be good to know
22:54:24 melwitt commit message explains why to do that too vs ignore DBDuplicateEntry
22:54:30 mnaser neat
22:54:56 melwitt oh it does? how do yall do discover_hosts? keep running until you receive status code 0? that's what tripleo does
22:55:39 melwitt the status code is a little weird on that command where non-zero means "there's more work to do" and zero means "I'm done"
22:55:54 melwitt and I think that's the review feedback I got on that patch currently is to enhance the doc to explain that
22:57:13 openstackgerrit Merged openstack/nova master: Stop handling cells v1 in '/os-servers' API https://review.openstack.org/651293
22:57:21 openstackgerrit Merged openstack/nova stable/rocky: Add functional recreate test for bug 1819963 https://review.openstack.org/648409
22:57:22 openstack bug 1819963 in OpenStack Compute (nova) rocky "Reverting a resize does not update the instance.availability_zone value to the source az" [Medium,In progress] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem)
22:57:36 openstackgerrit Merged openstack/nova stable/rocky: Update instance.availability_zone on revertResize https://review.openstack.org/648410
22:57:48 mnaser melwitt: we run discover_hosts every time once only on one of the conductors
22:58:03 mnaser I didn't know discover_hosts does work in batches
22:58:06 melwitt oh, so always expect to receive 0
22:58:29 melwitt I don't think it does. tripleo retries everything
22:58:43 mnaser https://github.com/openstack/openstack-ansible-os_nova/blob/d1aa5839b482c7412404ebb7bf050ee806fbf281/tasks/nova_db_post_setup.yml#L21-L26
22:59:06 mnaser ya I try to avoid, uh, brute forcing stuff :p
22:59:15 melwitt yeah it doesn't batch. it's just that non-zero means "I couldn't do the work I needed to do"
23:00:06 melwitt ok... so this will be a problem. we had been thinking to still return 1 if a collision happens i.e. not change the behavior
23:00:31 mnaser right, we can adapt our tasks easily then
23:00:40 melwitt but if you happen to run it while your scheduler does a discover_hosts, you will get a fail status code
23:00:54 mnaser yeah, our case is not parallel but racing with scheduler
23:01:02 melwitt yeah
23:01:42 melwitt ok, well if you want/need different behavior than is currently proposed, please add a comment on the patch review
23:02:01 mnaser ill add comments that we hit this
23:02:18 melwitt example: specific non-zero code that means "collision detected"
23:02:25 melwitt ok
23:05:55 mnaser melwitt: let a comment, danke :)
23:06:42 melwitt kewl
#openstack-nova - 2019-04-18
01:37:18 imacdonn anyone know how to trigger GMR when nova(-api) is running under uWSGI? Seems that uWSGI has its own interpretation of SIGUSR2 :/
03:26:25 openstackgerrit ya.wang proposed openstack/nova-specs master: Expose auto converge and post copy https://review.openstack.org/651681
05:12:50 openstackgerrit Merged openstack/nova master: Remove 'nova-manage cell' commands https://review.openstack.org/651294
05:12:57 openstackgerrit Merged openstack/nova master: Stop handling cells v1 for console authentication https://review.openstack.org/651295
05:45:48 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242
05:45:48 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.openstack.org/631243
05:45:49 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.openstack.org/631244
05:45:49 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.openstack.org/631245
05:58:48 openstackgerrit Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.openstack.org/651969
06:04:38 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.openstack.org/637716
06:38:28 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242
06:38:29 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.openstack.org/631243
06:38:29 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.openstack.org/631244
06:38:30 openstackgerrit Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.openstack.org/631245
07:31:58 openstackgerrit Boxiang Zhu proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645520
08:01:28 openstackgerrit Hamdy Khader proposed openstack/nova master: Do not perform port update in case of baremetal instance. https://review.openstack.org/649345
08:15:39 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Fix a description for config_drive parameter https://review.openstack.org/653683
09:04:47 openstackgerrit Surya Seetharaman proposed openstack/nova master: [WIP] Support adding the reason behind a server lock https://review.openstack.org/648662
09:04:47 openstackgerrit Surya Seetharaman proposed openstack/nova master: Plumbing for locking an instance with reason https://review.openstack.org/653691
09:14:47 openstackgerrit Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.openstack.org/649963
09:16:59 openstackgerrit Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.openstack.org/651969
09:19:24 plestang Hi all!
09:19:49 plestang I have a question regarding nova-conductor
09:20:29 plestang regurlarly compute hosts are requesting for unconfirmed migration by dest compute
09:21:07 plestang The function called is here: https://github.com/openstack/nova/blob/03322bb517925a9f5a04ebdb41c3fd31e7962440/nova/db/sqlalchemy/api.py#L4313
09:21:23 plestang the read_deleted params is set to yes
09:21:30 plestang in model_query
09:21:50 plestang it generates some load on our infra

Earlier   Later