| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-17 | |||
| 22:29:47 | mnaser | this is really weird | |
| 22:29:48 | mnaser | http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/ara-report/result/2a432934-7b70-491a-9571-a40e85cd15c6/ | |
| 22:30:05 | mnaser | "Checking host mapping for compute host" repeated twice, but with the same uuid? | |
| 22:31:21 | mnaser | the heck | |
| 22:31:22 | mnaser | http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-compute.log.txt.gz#_2019-04-17_21_52_21_727 | |
| 22:31:28 | mnaser | http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-compute.log.txt.gz#_2019-04-17_21_52_21_819 | |
| 22:32:17 | mnaser | in the same pid.. | |
| 22:33:42 | melwitt | are you running discover_hosts in parallel during a deployment? | |
| 22:34:59 | melwitt | https://bugs.launchpad.net/nova/+bug/1824445 | |
| 22:35:00 | openstack | Launchpad bug 1824445 in OpenStack Compute (nova) "nova-manage cellv2 discover_hosts traces when run in parallel" [Low,In progress] - Assigned to melanie witt (melwitt) | |
| 22:36:34 | mnaser | melwitt: nope, should be once only | |
| 22:36:55 | mnaser | but also is it normal that there is two log lines that say compute node created | |
| 22:37:58 | mnaser | melwitt: we also delegate_to the nova-conductor host so we run it once | |
| 22:39:05 | melwitt | I've only seen this before when the command is run in parallel presumably on different hosts | |
| 22:39:40 | melwitt | that's not normal, it shouldn't be trying to create the host mapping for the same host twice | |
| 22:39:52 | melwitt | I don't even get how that could happen non-parallel | |
| 22:39:58 | mnaser | yeah.. | |
| 22:40:05 | mnaser | and the pid is the same too | |
| 22:41:17 | melwitt | I dunno, seems like dark magic | |
| 22:42:09 | melwitt | hm, ok the last log is ringing a bell | |
| 22:42:36 | melwitt | did you delete a service via the API and then not restart the nova-compute that goes with it? | |
| 22:43:16 | melwitt | because nova-compute will only create a compute node record when it starts up, it will not recover if the compute node record gets deleted out from under it while it's running | |
| 22:46:19 | mnaser | we don't delete anything afaik :\ | |
| 22:46:27 | mnaser | during the CI run at least | |
| 22:46:51 | melwitt | ok. I don't see how that would be related to the discover_hosts thing anyway if it were happening | |
| 22:48:29 | mnaser | https://github.com/openstack/openstack-ansible-os_nova/blob/d1aa5839b482c7412404ebb7bf050ee806fbf281/tasks/main.yml#L233-L243 | |
| 22:48:42 | melwitt | mnaser: oh, sorry, I see now it's saying Checking first and then Creating, so that's probably normal | |
| 22:48:51 | mnaser | oh oops | |
| 22:48:54 | melwitt | I thought it was the same message twice but it's not | |
| 22:49:10 | mnaser | you're right yes | |
| 22:49:12 | mnaser | "Compute node record created" is just once | |
| 22:49:20 | melwitt | but the duplicate, I've only seen that when one nova-manage is racing against another. do you have the discover_hosts scheduler periodic enabled? | |
| 22:49:44 | mnaser | I think we enable that based on something | |
| 22:49:45 | mnaser | one sec | |
| 22:49:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.openstack.org/643664 | |
| 22:50:08 | melwitt | that's another possibility, if you were unlucky enough to have that fire at the same time as another scheduler, or if you were to run 'nova-manage cellv2 discover_hosts' at the same time a scheduler periodic discover_hosts fires | |
| 22:50:39 | mnaser | scheduler.discover_hosts_in_cells_interval = 60 | |
| 22:50:55 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899 | |
| 22:51:07 | mnaser | does discover_hosts run in the conductor I assume? | |
| 22:51:07 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899 | |
| 22:51:20 | melwitt | ok, and is that set on all schedulers or just one? (recommendation is to enable it on only one) see https://review.openstack.org/651947 | |
| 22:51:25 | melwitt | scheduler | |
| 22:52:25 | mnaser | http://logs.openstack.org/32/653532/1/gate/openstack-ansible-deploy-aio_metal-ubuntu-bionic/a2f00da/logs/host/nova/nova-scheduler.log.txt.gz#_2019-04-17_21_52_03_776 | |
| 22:52:30 | mnaser | yeah its around the same time | |
| 22:52:35 | mnaser | we have it enabled on all hosts :X | |
| 22:53:13 | melwitt | ok. yeah, I don't think we knew this was going to be a potential problem, or at least I didn't know | |
| 22:53:22 | mnaser | I mean its the first time I see this | |
| 22:53:24 | mnaser | but at least we kno wwhy | |
| 22:53:26 | melwitt | the trace isn't harmful, it just means discover_hosts collided with another one | |
| 22:53:43 | melwitt | so the patch I linked earlier aims to warn and give guidance instead of tracing | |
| 22:54:15 | mnaser | yeah that would be good because the ansible obviously fails when it sees that | |
| 22:54:18 | mnaser | so that would be good to know | |
| 22:54:24 | melwitt | commit message explains why to do that too vs ignore DBDuplicateEntry | |
| 22:54:30 | mnaser | neat | |
| 22:54:56 | melwitt | oh it does? how do yall do discover_hosts? keep running until you receive status code 0? that's what tripleo does | |
| 22:55:39 | melwitt | the status code is a little weird on that command where non-zero means "there's more work to do" and zero means "I'm done" | |
| 22:55:54 | melwitt | and I think that's the review feedback I got on that patch currently is to enhance the doc to explain that | |
| 22:57:13 | openstackgerrit | Merged openstack/nova master: Stop handling cells v1 in '/os-servers' API https://review.openstack.org/651293 | |
| 22:57:21 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional recreate test for bug 1819963 https://review.openstack.org/648409 | |
| 22:57:22 | openstack | bug 1819963 in OpenStack Compute (nova) rocky "Reverting a resize does not update the instance.availability_zone value to the source az" [Medium,In progress] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem) | |
| 22:57:36 | openstackgerrit | Merged openstack/nova stable/rocky: Update instance.availability_zone on revertResize https://review.openstack.org/648410 | |
| 22:57:48 | mnaser | melwitt: we run discover_hosts every time once only on one of the conductors | |
| 22:58:03 | mnaser | I didn't know discover_hosts does work in batches | |
| 22:58:06 | melwitt | oh, so always expect to receive 0 | |
| 22:58:29 | melwitt | I don't think it does. tripleo retries everything | |
| 22:58:43 | mnaser | https://github.com/openstack/openstack-ansible-os_nova/blob/d1aa5839b482c7412404ebb7bf050ee806fbf281/tasks/nova_db_post_setup.yml#L21-L26 | |
| 22:59:06 | mnaser | ya I try to avoid, uh, brute forcing stuff :p | |
| 22:59:15 | melwitt | yeah it doesn't batch. it's just that non-zero means "I couldn't do the work I needed to do" | |
| 23:00:06 | melwitt | ok... so this will be a problem. we had been thinking to still return 1 if a collision happens i.e. not change the behavior | |
| 23:00:31 | mnaser | right, we can adapt our tasks easily then | |
| 23:00:40 | melwitt | but if you happen to run it while your scheduler does a discover_hosts, you will get a fail status code | |
| 23:00:54 | mnaser | yeah, our case is not parallel but racing with scheduler | |
| 23:01:02 | melwitt | yeah | |
| 23:01:42 | melwitt | ok, well if you want/need different behavior than is currently proposed, please add a comment on the patch review | |
| 23:02:01 | mnaser | ill add comments that we hit this | |
| 23:02:18 | melwitt | example: specific non-zero code that means "collision detected" | |
| 23:02:25 | melwitt | ok | |
| 23:05:55 | mnaser | melwitt: let a comment, danke :) | |
| 23:06:42 | melwitt | kewl | |
| #openstack-nova - 2019-04-18 | |||
| 01:37:18 | imacdonn | anyone know how to trigger GMR when nova(-api) is running under uWSGI? Seems that uWSGI has its own interpretation of SIGUSR2 :/ | |
| 03:26:25 | openstackgerrit | ya.wang proposed openstack/nova-specs master: Expose auto converge and post copy https://review.openstack.org/651681 | |
| 05:12:50 | openstackgerrit | Merged openstack/nova master: Remove 'nova-manage cell' commands https://review.openstack.org/651294 | |
| 05:12:57 | openstackgerrit | Merged openstack/nova master: Stop handling cells v1 for console authentication https://review.openstack.org/651295 | |
| 05:45:48 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.openstack.org/631243 | |
| 05:45:48 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242 | |
| 05:45:49 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.openstack.org/631245 | |
| 05:45:49 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.openstack.org/631244 | |
| 05:58:48 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.openstack.org/651969 | |
| 06:04:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens https://review.openstack.org/637716 | |
| 06:38:28 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242 | |
| 06:38:29 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.openstack.org/631244 | |
| 06:38:29 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.openstack.org/631243 | |
| 06:38:30 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.openstack.org/631245 | |
| 07:31:58 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645520 | |
| 08:01:28 | openstackgerrit | Hamdy Khader proposed openstack/nova master: Do not perform port update in case of baremetal instance. https://review.openstack.org/649345 | |
| 08:15:39 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Fix a description for config_drive parameter https://review.openstack.org/653683 | |
| 09:04:47 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Plumbing for locking an instance with reason https://review.openstack.org/653691 | |
| 09:04:47 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: [WIP] Support adding the reason behind a server lock https://review.openstack.org/648662 | |
| 09:14:47 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.openstack.org/649963 | |
| 09:16:59 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.openstack.org/651969 | |