| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 23:39:52 | sbezverk | mriedem: any known issue with placement in ocata you are aware? | |
| 23:39:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix functional regression/recreate test for bug 1671648 https://review.openstack.org/446685 | |
| 23:39:56 | openstack | bug 1671648 in OpenStack Compute (nova) ocata "Instances are not rescheduled after deploy fails" [High,In progress] https://launchpad.net/bugs/1671648 - Assigned to Matt Riedemann (mriedem) | |
| 23:40:21 | mriedem | sbezverk: nothing major for placement that i can think of | |
| 23:40:33 | mriedem | sbezverk: we did release some fixes for ocata in 15.0.1 earlier in the week, | |
| 23:40:39 | mriedem | so if you're upgrading, i'd use 15.0.1 | |
| 23:41:13 | sbezverk | mriedem: I brining up at the gate ocata images in kolla-kubernetes project | |
| 23:41:41 | sbezverk | what worked with stable/newton seems broken in ocata, but it might be our issue too.. | |
| 23:42:57 | mriedem | sbezverk: i guess i'd need a more specific example of what's not working | |
| 23:43:56 | Sukhdev | dear Nova folks, after fresh install of devstack, Nova continue to return "No Valid host found" error - can somebody provide some wisdom, please? See the details here - http://paste.openstack.org/show/603034/ | |
| 23:44:29 | sbezverk | mriedem: :) glad you asked. When I start instance I get " Placement API service is not responding." in nova scheduler log | |
| 23:44:59 | mriedem | sbezverk: are you starting the placement service before the nova-scheduler service? | |
| 23:45:04 | sbezverk | mriedem: it looks like connectivity issue but nothing change other than just version of images | |
| 23:45:12 | mriedem | is there a "placement" endpoint in the service catalog? | |
| 23:45:37 | sbezverk | mriedem: yep, service catalog shows placement entries | |
| 23:45:38 | mriedem | is nova.conf for the scheduler configured to use placement in the [placement] section, and are the credentials in there correct? | |
| 23:46:13 | sbezverk | mriedem: all valid points, but as I said nothing changed in the gate job other than image version | |
| 23:46:30 | mriedem | the scheduler didn't use placement in newton, | |
| 23:46:34 | mriedem | so that's a main different with ocata | |
| 23:46:38 | sbezverk | mriedem: was there any change in endpoint registrations between newton and ocata? | |
| 23:46:46 | mriedem | in newton, only the nova-compute service tried connecting to the placement service | |
| 23:46:55 | mriedem | no there was no change in endpoint registration | |
| 23:47:04 | sbezverk | mriedem: AHHH I did not know that | |
| 23:47:17 | mriedem | i'd compare the nova.conf [placement] sections between your nova-compute and nova-scheduler services | |
| 23:47:33 | mriedem | make sure that whatever you did to make it work for nova-compute, that you do that for nova-scheduler too | |
| 23:47:55 | mriedem | sbezverk: https://docs.openstack.org/developer/nova/placement.html#upgrade-notes in case you haven't seen that yet | |
| 23:48:04 | sbezverk | mriedem: if you have 2 minutes I can get it, would be great if you cold take a look at it | |
| 23:48:21 | mriedem | oh boy | |
| 23:48:23 | mriedem | :) | |
| 23:48:42 | mriedem | Sukhdev: is the nova-compute service running and listed as up when you run "nova service-list"? | |
| 23:49:24 | Kevin_Zheng | mriedem: yeah thanks | |
| 23:49:50 | mriedem | Kevin_Zheng: yeah as in you agree with the changes, or yeah as in you will look? | |
| 23:50:10 | Sukhdev | mriedem : yes - see here - http://paste.openstack.org/show/603035/ | |
| 23:50:34 | Kevin_Zheng | agree：） | |
| 23:51:13 | sbezverk | mriedem: http://paste.openstack.org/show/603036/ | |
| 23:51:23 | sbezverk | mriedem: look identical to me :( | |
| 23:52:12 | Kevin_Zheng | thanks alot | |
| 23:53:17 | Sukhdev | mriedem : anything else that I should look for? BTW, I am running the latest code (did git pull 5-10 days ago) | |
| 23:56:24 | mriedem | Kevin_Zheng: yw | |
| 23:56:48 | mriedem | Sukhdev: check the nova-compute logs for errors | |
| 23:57:22 | mriedem | sbezverk: those look the same to me too | |
| 23:57:32 | mriedem | sbezverk: the services all run in individual containers right? | |
| 23:58:39 | mriedem | is there anything in the deployment tooling that has to do anything specific for the compute service wrt placement that the scheduler container isn't doing? | |
| #openstack-nova - 2017-03-17 | |||
| 00:03:08 | sbezverk | mriedem: correct about individual containers | |
| 00:03:42 | sbezverk | mriedem: in paralel I have a job running newton images and it passes wihtout issues | |
| 00:04:19 | sbezverk | mriedem: I will continue debugging to confirm connectivity issue.. curl should tell me that | |
| 00:04:52 | Sukhdev | mriedem : I see lots of time out errors in n-cpu.logs - here is a sample - http://paste.openstack.org/show/603037/ | |
| 00:09:25 | mriedem | Sukhdev: do you have any customizations to the local.conf you used to stack devstack? can you post your local.conf in a paste? | |
| 00:09:44 | mriedem | seems you're having rpc issues, which would mean investigating the oslo.messaging configuration values in nova.conf | |
| 00:13:30 | Sukhdev | mriedem : here is local.conf - http://paste.openstack.org/show/603039/ | |
| 00:13:33 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446777 | |
| 00:15:18 | Sukhdev | mriedem : here is /etc/nova/nova.conf - http://paste.openstack.org/show/603040/ | |
| 00:22:10 | Sukhdev | mriedem : can you spot anything suspicious in either of those configs? | |
| 00:28:04 | mriedem | they look ok, | |
| 00:28:20 | mriedem | they are all for ironic though, and i've never setup an ironic devstack instance, so i'm not sure what to be looking for, | |
| 00:28:28 | mriedem | you might want to ask in the #openstack-ironic channel | |
| 00:31:52 | openstackgerrit | Tony Breeds proposed openstack/os-vif master: Add Constraints support https://review.openstack.org/413325 | |
| 00:38:02 | Sukhdev | mriedem : this does not go to ironic at all - as soon as nova boot is issued, it fails in nova itself. So, there is no log in the Ironic side | |
| 00:42:36 | mriedem | Sukhdev: is ironic running OK? | |
| 00:42:51 | mriedem | the ironic virt driver in the nova-compute service will be getting resource information from the ironic API service | |
| 00:42:51 | Sukhdev | mriedem : yes | |
| 00:42:58 | mriedem | that resource information is fed into the placement service, | |
| 00:43:09 | mriedem | and the filter scheduler uses that information for scheduling decisions | |
| 00:44:05 | Sukhdev | mriedem : I did not look into Ironic API logs - let me take a peek | |
| 00:46:31 | mriedem | Sukhdev: you could also compare to the local.conf from the ironic CI job that runs on nova changes here http://logs.openstack.org/01/446601/1/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/4630b48/logs/local.conf.txt.gz | |
| 00:47:43 | mriedem | it could be the flavors you have defined don't fit the nodes available | |
| 00:47:57 | Sukhdev | mriedem : excellent Idea - thanks for the pointer - let me check if some of this config is causing the issue. | |
| 00:53:14 | Sukhdev | mriedem : One quick question (for my clarification) I read somewhere while looking for help on Google it stated that in Ocata release Nova mendate cell configuration - is it true? Do I need to add anything in local.conf in this regard> | |
| 00:54:09 | mriedem | Sukhdev: you don't need anything in local.conf for cells v2 in ocata | |
| 00:54:43 | Sukhdev | mriedem : cool - thanks - was worried if that has anything to do with this :-) | |
| 00:55:03 | mriedem | i don't think it would, the scheduler in ocata didn't take cells into consideration | |
| 00:55:35 | Sukhdev | Great - thanks | |
| 00:55:41 | sbezverk | mriedemЬ curl returns 401 Unauthorized | |
| 00:56:53 | sbezverk | mriedem: it does not look like network issue then.. | |
| 00:57:40 | mriedem | sbezverk: did you provide a token in the curl request to / ? | |
| 01:00:47 | sbezverk | mriedem: cannot do it as it is a gate job | |
| 01:01:01 | melwitt | mriedem: argh, dangit (test trace) | |
| 01:01:22 | sbezverk | mriedem: do not have much flexebility there | |
| 01:01:24 | melwitt | oh, you took care of it. thanks | |
| 01:01:27 | mriedem | melwitt: yup | |
| 01:02:12 | mriedem | sbezverk: i'm not sure what you mean. you're scripting this into the CI job right? if you've added something that makes a curl request to the root of the placement endpoint from the nova-scheduler container, can't you also get a token and include that in the request, from within the script/CI job? | |
| 01:02:31 | mriedem | OS_TOKEN=$(openstack token issue -f value -c id) | |
| 01:03:35 | sbezverk | mriedem: cool thanks, let me try with it | |
| 01:08:59 | dtp-afk | jroll are you still here? | |
| 01:10:59 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add api-ref for filter/sort whitelist https://review.openstack.org/421760 | |
| 01:18:05 | openstackgerrit | Dan Peschman proposed openstack/nova master: Option to always create network metadata information as static https://review.openstack.org/336293 | |
| 01:39:28 | sbezverk | mriedem: ok so the process is alive, there must be something with scheduler preventing it finding it.. | |
| 01:39:54 | sbezverk | mriedem: http://paste.openstack.org/show/603043/ | |
| 01:40:34 | mriedem | sbezverk: try restarting nova-scheduler, i think we cache the failed attempt and don't try again for like 10 minutes | |
| 01:44:07 | dtp | mriedem, if i have other ideas on implementation of https://blueprints.launchpad.net/nova/+spec/service-hyper-pci-uuid-in-api, should i just keep them to myself? | |
| 01:44:28 | sbezverk | mriedem: well, it is at the gate. once failed the job is failed/gone.. | |
| 01:45:32 | mriedem | sbezverk: you can't reproduce locally? | |
| 01:45:37 | mriedem | dtp: umm, idk | |
| 01:45:46 | mriedem | dtp: we could talk tomorrow, i've got to call it a day | |
| 01:45:56 | dtp | sounds good | |
| 01:46:04 | sbezverk | mriedem: when scheduler starts it tries to communicate with placement right away? | |
| 01:46:21 | mriedem | sbezverk: i don't think so no | |
| 01:46:30 | mriedem | only once a request to create an instance comes in | |
| 01:46:44 | sbezverk | mriedem: ok so ordering would not matter much | |
| 01:46:52 | sbezverk | I mean process start up | |