| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 23:20:01 | mriedem | yeah you need to use fake.restore_nodes | |
| 23:24:09 | sbezverk | hello, I need to check liveness of nova placement api process, could you recommen any methods to check not just that the socket is opened but also that the process is alive and sane. | |
| 23:24:26 | mriedem | sbezverk: make a curl request to / ? | |
| 23:24:54 | mriedem | that's testing much more than just the process is running | |
| 23:25:01 | sbezverk | mriedem: would it not just check apachie liveness? | |
| 23:25:28 | mriedem | making a curl request to the actual endpoint's root would provide the versions available, which means it's up and running and accepting requests | |
| 23:25:31 | mriedem | and serving a response | |
| 23:25:51 | mriedem | i don't know if a token is required for GET / on the placement endpoint though | |
| 23:26:22 | sbezverk | mriedem: cool, thank you very much. I was not sure if curling for root is sufficient | |
| 23:26:53 | mriedem | yup that will get you the versions available | |
| 23:39:19 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add api-ref for filter/sort whitelist https://review.openstack.org/421760 | |
| 23:39:52 | sbezverk | mriedem: any known issue with placement in ocata you are aware? | |
| 23:39:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix functional regression/recreate test for bug 1671648 https://review.openstack.org/446685 | |
| 23:39:56 | openstack | bug 1671648 in OpenStack Compute (nova) ocata "Instances are not rescheduled after deploy fails" [High,In progress] https://launchpad.net/bugs/1671648 - Assigned to Matt Riedemann (mriedem) | |
| 23:40:21 | mriedem | sbezverk: nothing major for placement that i can think of | |
| 23:40:33 | mriedem | sbezverk: we did release some fixes for ocata in 15.0.1 earlier in the week, | |
| 23:40:39 | mriedem | so if you're upgrading, i'd use 15.0.1 | |
| 23:41:13 | sbezverk | mriedem: I brining up at the gate ocata images in kolla-kubernetes project | |
| 23:41:41 | sbezverk | what worked with stable/newton seems broken in ocata, but it might be our issue too.. | |
| 23:42:57 | mriedem | sbezverk: i guess i'd need a more specific example of what's not working | |
| 23:43:56 | Sukhdev | dear Nova folks, after fresh install of devstack, Nova continue to return "No Valid host found" error - can somebody provide some wisdom, please? See the details here - http://paste.openstack.org/show/603034/ | |
| 23:44:29 | sbezverk | mriedem: :) glad you asked. When I start instance I get " Placement API service is not responding." in nova scheduler log | |
| 23:44:59 | mriedem | sbezverk: are you starting the placement service before the nova-scheduler service? | |
| 23:45:04 | sbezverk | mriedem: it looks like connectivity issue but nothing change other than just version of images | |
| 23:45:12 | mriedem | is there a "placement" endpoint in the service catalog? | |
| 23:45:37 | sbezverk | mriedem: yep, service catalog shows placement entries | |
| 23:45:38 | mriedem | is nova.conf for the scheduler configured to use placement in the [placement] section, and are the credentials in there correct? | |
| 23:46:13 | sbezverk | mriedem: all valid points, but as I said nothing changed in the gate job other than image version | |
| 23:46:30 | mriedem | the scheduler didn't use placement in newton, | |
| 23:46:34 | mriedem | so that's a main different with ocata | |
| 23:46:38 | sbezverk | mriedem: was there any change in endpoint registrations between newton and ocata? | |
| 23:46:46 | mriedem | in newton, only the nova-compute service tried connecting to the placement service | |
| 23:46:55 | mriedem | no there was no change in endpoint registration | |
| 23:47:04 | sbezverk | mriedem: AHHH I did not know that | |
| 23:47:17 | mriedem | i'd compare the nova.conf [placement] sections between your nova-compute and nova-scheduler services | |
| 23:47:33 | mriedem | make sure that whatever you did to make it work for nova-compute, that you do that for nova-scheduler too | |
| 23:47:55 | mriedem | sbezverk: https://docs.openstack.org/developer/nova/placement.html#upgrade-notes in case you haven't seen that yet | |
| 23:48:04 | sbezverk | mriedem: if you have 2 minutes I can get it, would be great if you cold take a look at it | |
| 23:48:21 | mriedem | oh boy | |
| 23:48:23 | mriedem | :) | |
| 23:48:42 | mriedem | Sukhdev: is the nova-compute service running and listed as up when you run "nova service-list"? | |
| 23:49:24 | Kevin_Zheng | mriedem: yeah thanks | |
| 23:49:50 | mriedem | Kevin_Zheng: yeah as in you agree with the changes, or yeah as in you will look? | |
| 23:50:10 | Sukhdev | mriedem : yes - see here - http://paste.openstack.org/show/603035/ | |
| 23:50:34 | Kevin_Zheng | agree：） | |
| 23:51:13 | sbezverk | mriedem: http://paste.openstack.org/show/603036/ | |
| 23:51:23 | sbezverk | mriedem: look identical to me :( | |
| 23:52:12 | Kevin_Zheng | thanks alot | |
| 23:53:17 | Sukhdev | mriedem : anything else that I should look for? BTW, I am running the latest code (did git pull 5-10 days ago) | |
| 23:56:24 | mriedem | Kevin_Zheng: yw | |
| 23:56:48 | mriedem | Sukhdev: check the nova-compute logs for errors | |
| 23:57:22 | mriedem | sbezverk: those look the same to me too | |
| 23:57:32 | mriedem | sbezverk: the services all run in individual containers right? | |
| 23:58:39 | mriedem | is there anything in the deployment tooling that has to do anything specific for the compute service wrt placement that the scheduler container isn't doing? | |
| #openstack-nova - 2017-03-17 | |||
| 00:03:08 | sbezverk | mriedem: correct about individual containers | |
| 00:03:42 | sbezverk | mriedem: in paralel I have a job running newton images and it passes wihtout issues | |
| 00:04:19 | sbezverk | mriedem: I will continue debugging to confirm connectivity issue.. curl should tell me that | |
| 00:04:52 | Sukhdev | mriedem : I see lots of time out errors in n-cpu.logs - here is a sample - http://paste.openstack.org/show/603037/ | |
| 00:09:25 | mriedem | Sukhdev: do you have any customizations to the local.conf you used to stack devstack? can you post your local.conf in a paste? | |
| 00:09:44 | mriedem | seems you're having rpc issues, which would mean investigating the oslo.messaging configuration values in nova.conf | |
| 00:13:30 | Sukhdev | mriedem : here is local.conf - http://paste.openstack.org/show/603039/ | |
| 00:13:33 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446777 | |
| 00:15:18 | Sukhdev | mriedem : here is /etc/nova/nova.conf - http://paste.openstack.org/show/603040/ | |
| 00:22:10 | Sukhdev | mriedem : can you spot anything suspicious in either of those configs? | |
| 00:28:04 | mriedem | they look ok, | |
| 00:28:20 | mriedem | they are all for ironic though, and i've never setup an ironic devstack instance, so i'm not sure what to be looking for, | |
| 00:28:28 | mriedem | you might want to ask in the #openstack-ironic channel | |
| 00:31:52 | openstackgerrit | Tony Breeds proposed openstack/os-vif master: Add Constraints support https://review.openstack.org/413325 | |
| 00:38:02 | Sukhdev | mriedem : this does not go to ironic at all - as soon as nova boot is issued, it fails in nova itself. So, there is no log in the Ironic side | |
| 00:42:36 | mriedem | Sukhdev: is ironic running OK? | |
| 00:42:51 | mriedem | the ironic virt driver in the nova-compute service will be getting resource information from the ironic API service | |
| 00:42:51 | Sukhdev | mriedem : yes | |
| 00:42:58 | mriedem | that resource information is fed into the placement service, | |
| 00:43:09 | mriedem | and the filter scheduler uses that information for scheduling decisions | |
| 00:44:05 | Sukhdev | mriedem : I did not look into Ironic API logs - let me take a peek | |
| 00:46:31 | mriedem | Sukhdev: you could also compare to the local.conf from the ironic CI job that runs on nova changes here http://logs.openstack.org/01/446601/1/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/4630b48/logs/local.conf.txt.gz | |
| 00:47:43 | mriedem | it could be the flavors you have defined don't fit the nodes available | |
| 00:47:57 | Sukhdev | mriedem : excellent Idea - thanks for the pointer - let me check if some of this config is causing the issue. | |
| 00:53:14 | Sukhdev | mriedem : One quick question (for my clarification) I read somewhere while looking for help on Google it stated that in Ocata release Nova mendate cell configuration - is it true? Do I need to add anything in local.conf in this regard> | |
| 00:54:09 | mriedem | Sukhdev: you don't need anything in local.conf for cells v2 in ocata | |
| 00:54:43 | Sukhdev | mriedem : cool - thanks - was worried if that has anything to do with this :-) | |
| 00:55:03 | mriedem | i don't think it would, the scheduler in ocata didn't take cells into consideration | |
| 00:55:35 | Sukhdev | Great - thanks | |
| 00:55:41 | sbezverk | mriedemЬ curl returns 401 Unauthorized | |
| 00:56:53 | sbezverk | mriedem: it does not look like network issue then.. | |
| 00:57:40 | mriedem | sbezverk: did you provide a token in the curl request to / ? | |
| 01:00:47 | sbezverk | mriedem: cannot do it as it is a gate job | |
| 01:01:01 | melwitt | mriedem: argh, dangit (test trace) | |
| 01:01:22 | sbezverk | mriedem: do not have much flexebility there | |
| 01:01:24 | melwitt | oh, you took care of it. thanks | |
| 01:01:27 | mriedem | melwitt: yup | |
| 01:02:12 | mriedem | sbezverk: i'm not sure what you mean. you're scripting this into the CI job right? if you've added something that makes a curl request to the root of the placement endpoint from the nova-scheduler container, can't you also get a token and include that in the request, from within the script/CI job? | |
| 01:02:31 | mriedem | OS_TOKEN=$(openstack token issue -f value -c id) | |
| 01:03:35 | sbezverk | mriedem: cool thanks, let me try with it | |
| 01:08:59 | dtp-afk | jroll are you still here? | |
| 01:10:59 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add api-ref for filter/sort whitelist https://review.openstack.org/421760 | |
| 01:18:05 | openstackgerrit | Dan Peschman proposed openstack/nova master: Option to always create network metadata information as static https://review.openstack.org/336293 | |
| 01:39:28 | sbezverk | mriedem: ok so the process is alive, there must be something with scheduler preventing it finding it.. | |
| 01:39:54 | sbezverk | mriedem: http://paste.openstack.org/show/603043/ | |
| 01:40:34 | mriedem | sbezverk: try restarting nova-scheduler, i think we cache the failed attempt and don't try again for like 10 minutes | |