| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 23:47:04 | sbezverk | mriedem: AHHH I did not know that | |
| 23:47:17 | mriedem | i'd compare the nova.conf [placement] sections between your nova-compute and nova-scheduler services | |
| 23:47:33 | mriedem | make sure that whatever you did to make it work for nova-compute, that you do that for nova-scheduler too | |
| 23:47:55 | mriedem | sbezverk: https://docs.openstack.org/developer/nova/placement.html#upgrade-notes in case you haven't seen that yet | |
| 23:48:04 | sbezverk | mriedem: if you have 2 minutes I can get it, would be great if you cold take a look at it | |
| 23:48:21 | mriedem | oh boy | |
| 23:48:23 | mriedem | :) | |
| 23:48:42 | mriedem | Sukhdev: is the nova-compute service running and listed as up when you run "nova service-list"? | |
| 23:49:24 | Kevin_Zheng | mriedem: yeah thanks | |
| 23:49:50 | mriedem | Kevin_Zheng: yeah as in you agree with the changes, or yeah as in you will look? | |
| 23:50:10 | Sukhdev | mriedem : yes - see here - http://paste.openstack.org/show/603035/ | |
| 23:50:34 | Kevin_Zheng | agree：） | |
| 23:51:13 | sbezverk | mriedem: http://paste.openstack.org/show/603036/ | |
| 23:51:23 | sbezverk | mriedem: look identical to me :( | |
| 23:52:12 | Kevin_Zheng | thanks alot | |
| 23:53:17 | Sukhdev | mriedem : anything else that I should look for? BTW, I am running the latest code (did git pull 5-10 days ago) | |
| 23:56:24 | mriedem | Kevin_Zheng: yw | |
| 23:56:48 | mriedem | Sukhdev: check the nova-compute logs for errors | |
| 23:57:22 | mriedem | sbezverk: those look the same to me too | |
| 23:57:32 | mriedem | sbezverk: the services all run in individual containers right? | |
| 23:58:39 | mriedem | is there anything in the deployment tooling that has to do anything specific for the compute service wrt placement that the scheduler container isn't doing? | |
| #openstack-nova - 2017-03-17 | |||
| 00:03:08 | sbezverk | mriedem: correct about individual containers | |
| 00:03:42 | sbezverk | mriedem: in paralel I have a job running newton images and it passes wihtout issues | |
| 00:04:19 | sbezverk | mriedem: I will continue debugging to confirm connectivity issue.. curl should tell me that | |
| 00:04:52 | Sukhdev | mriedem : I see lots of time out errors in n-cpu.logs - here is a sample - http://paste.openstack.org/show/603037/ | |
| 00:09:25 | mriedem | Sukhdev: do you have any customizations to the local.conf you used to stack devstack? can you post your local.conf in a paste? | |
| 00:09:44 | mriedem | seems you're having rpc issues, which would mean investigating the oslo.messaging configuration values in nova.conf | |
| 00:13:30 | Sukhdev | mriedem : here is local.conf - http://paste.openstack.org/show/603039/ | |
| 00:13:33 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446777 | |
| 00:15:18 | Sukhdev | mriedem : here is /etc/nova/nova.conf - http://paste.openstack.org/show/603040/ | |
| 00:22:10 | Sukhdev | mriedem : can you spot anything suspicious in either of those configs? | |
| 00:28:04 | mriedem | they look ok, | |
| 00:28:20 | mriedem | they are all for ironic though, and i've never setup an ironic devstack instance, so i'm not sure what to be looking for, | |
| 00:28:28 | mriedem | you might want to ask in the #openstack-ironic channel | |
| 00:31:52 | openstackgerrit | Tony Breeds proposed openstack/os-vif master: Add Constraints support https://review.openstack.org/413325 | |
| 00:38:02 | Sukhdev | mriedem : this does not go to ironic at all - as soon as nova boot is issued, it fails in nova itself. So, there is no log in the Ironic side | |
| 00:42:36 | mriedem | Sukhdev: is ironic running OK? | |
| 00:42:51 | Sukhdev | mriedem : yes | |
| 00:42:51 | mriedem | the ironic virt driver in the nova-compute service will be getting resource information from the ironic API service | |
| 00:42:58 | mriedem | that resource information is fed into the placement service, | |
| 00:43:09 | mriedem | and the filter scheduler uses that information for scheduling decisions | |
| 00:44:05 | Sukhdev | mriedem : I did not look into Ironic API logs - let me take a peek | |
| 00:46:31 | mriedem | Sukhdev: you could also compare to the local.conf from the ironic CI job that runs on nova changes here http://logs.openstack.org/01/446601/1/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/4630b48/logs/local.conf.txt.gz | |
| 00:47:43 | mriedem | it could be the flavors you have defined don't fit the nodes available | |
| 00:47:57 | Sukhdev | mriedem : excellent Idea - thanks for the pointer - let me check if some of this config is causing the issue. | |
| 00:53:14 | Sukhdev | mriedem : One quick question (for my clarification) I read somewhere while looking for help on Google it stated that in Ocata release Nova mendate cell configuration - is it true? Do I need to add anything in local.conf in this regard> | |
| 00:54:09 | mriedem | Sukhdev: you don't need anything in local.conf for cells v2 in ocata | |
| 00:54:43 | Sukhdev | mriedem : cool - thanks - was worried if that has anything to do with this :-) | |
| 00:55:03 | mriedem | i don't think it would, the scheduler in ocata didn't take cells into consideration | |
| 00:55:35 | Sukhdev | Great - thanks | |
| 00:55:41 | sbezverk | mriedemЬ curl returns 401 Unauthorized | |
| 00:56:53 | sbezverk | mriedem: it does not look like network issue then.. | |
| 00:57:40 | mriedem | sbezverk: did you provide a token in the curl request to / ? | |
| 01:00:47 | sbezverk | mriedem: cannot do it as it is a gate job | |
| 01:01:01 | melwitt | mriedem: argh, dangit (test trace) | |
| 01:01:22 | sbezverk | mriedem: do not have much flexebility there | |
| 01:01:24 | melwitt | oh, you took care of it. thanks | |
| 01:01:27 | mriedem | melwitt: yup | |
| 01:02:12 | mriedem | sbezverk: i'm not sure what you mean. you're scripting this into the CI job right? if you've added something that makes a curl request to the root of the placement endpoint from the nova-scheduler container, can't you also get a token and include that in the request, from within the script/CI job? | |
| 01:02:31 | mriedem | OS_TOKEN=$(openstack token issue -f value -c id) | |
| 01:03:35 | sbezverk | mriedem: cool thanks, let me try with it | |
| 01:08:59 | dtp-afk | jroll are you still here? | |
| 01:10:59 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add api-ref for filter/sort whitelist https://review.openstack.org/421760 | |
| 01:18:05 | openstackgerrit | Dan Peschman proposed openstack/nova master: Option to always create network metadata information as static https://review.openstack.org/336293 | |
| 01:39:28 | sbezverk | mriedem: ok so the process is alive, there must be something with scheduler preventing it finding it.. | |
| 01:39:54 | sbezverk | mriedem: http://paste.openstack.org/show/603043/ | |
| 01:40:34 | mriedem | sbezverk: try restarting nova-scheduler, i think we cache the failed attempt and don't try again for like 10 minutes | |
| 01:44:07 | dtp | mriedem, if i have other ideas on implementation of https://blueprints.launchpad.net/nova/+spec/service-hyper-pci-uuid-in-api, should i just keep them to myself? | |
| 01:44:28 | sbezverk | mriedem: well, it is at the gate. once failed the job is failed/gone.. | |
| 01:45:32 | mriedem | sbezverk: you can't reproduce locally? | |
| 01:45:37 | mriedem | dtp: umm, idk | |
| 01:45:46 | mriedem | dtp: we could talk tomorrow, i've got to call it a day | |
| 01:45:56 | dtp | sounds good | |
| 01:46:04 | sbezverk | mriedem: when scheduler starts it tries to communicate with placement right away? | |
| 01:46:21 | mriedem | sbezverk: i don't think so no | |
| 01:46:30 | mriedem | only once a request to create an instance comes in | |
| 01:46:44 | sbezverk | mriedem: ok so ordering would not matter much | |
| 01:46:52 | sbezverk | I mean process start up | |
| 01:47:17 | mriedem | sbezverk: placement should come before the scheduler, but i'd have to dig into details, which i can't right now | |
| 01:47:31 | mriedem | sbezverk: check that nova devref doc on placement that i linked earlier, there might be something i'm forgeting | |
| 01:47:56 | sbezverk | mriedem: ok I will, also will try to repro in local test bed. thank you for your help | |
| 01:48:14 | mriedem | yw | |
| 01:59:14 | gmann | alex_xu: sfinucan mriedem can you check this. updated enum value format (api-ref for sort/filter) - https://review.openstack.org/#/c/421760/ | |
| 02:23:15 | alex_xu | gmann: thanks, I will check it soon | |
| 02:24:11 | gmann | mriedem: alex_xu johnthetubaguy sfinucan snikitin : i doubt cpu id should be integer, i replied on https://review.openstack.org/#/c/445855/1 and https://review.openstack.org/#/c/434745/3 | |
| 02:24:20 | gmann | need your opinion | |
| 02:24:24 | gmann | alex_xu: thanks | |
| 02:26:12 | gmann | libvirt driver(current diagnostics info provider) seems not appending the CPU id | |
| 04:37:04 | gmann | BobBall: can we use VM_metrics.get_VCPUs_utilisation to fetch the CPU utilization ? - https://review.openstack.org/#/c/399613/29 | |
| 06:49:10 | alex_xu | gmann: may I say 'valid value' instead of 'enum value' at https://review.openstack.org/#/c/421760/15/api-ref/source/parameters.yaml@421 | |
| 06:49:30 | alex_xu | gmann: due to from the schema, it just a field accept any string | |
| 06:49:59 | alex_xu | gmann: you won't get error when you input something out of 'auto' and 'manual' | |
| 06:52:18 | gmann | alex_xu: ah, nice point. ll update. | |
| 06:52:26 | alex_xu | gmann: thanks | |
| 06:52:30 | gmann | alex_xu: please -1 :) | |
| 06:52:43 | alex_xu | gmann: ah, yea, got one point | |
| 06:53:50 | alex_xu | gmann: done, thanks :) | |
| 06:54:05 | alex_xu | gmann: I can update the patch online | |
| 06:54:15 | alex_xu | do you want me do that, and I can +2 directly | |
| 06:54:36 | gmann | alex_xu: updating | |