| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-02 | |||
| 13:14:08 | kashyap | Can anyone remind me again, has upstream Git master opened for Train, yet? (/me is dazed after PTO) | |
| 13:14:50 | sean-k-mooney | yes | |
| 13:14:55 | sean-k-mooney | like a week or two ago | |
| 13:15:03 | kashyap | Thanks | |
| 13:15:09 | sean-k-mooney | it opens after RC1 but we dont then to merge big thing for a few weeks after | |
| 13:17:29 | gibi_off | sean-k-mooney: for notifications we only emit the latest version only | |
| 13:17:41 | gibi_off | sean-k-mooney: so please update the samepl file according to the change in the object | |
| 13:18:00 | gibi_off | sean-k-mooney: the tests are functional ast they call the nova API and assert if notifications are received | |
| 13:18:52 | sean-k-mooney | gibi_off: ok but would they not avoid this issue if they parsed the notificaion into a python object and then checked that | |
| 13:19:16 | sean-k-mooney | gibi_off: also if your off today your doing it wrong :) but thanks ill update it | |
| 13:19:41 | kashyap | efried: When you get a moment, since Train has forked, might want to put this through: https://review.openstack.org/#/c/641981/ | |
| 13:20:24 | efried | kashyap: done, thanks for the reminder. | |
| 13:21:08 | gibi_off | sean-k-mooney: the parsed object would have different version | |
| 13:21:42 | gibi_off | sean-k-mooney: also nova cannot assume how the notifications are parsed by the consumer so we assert the json we emmit | |
| 13:21:51 | sean-k-mooney | ah good points | |
| 13:22:08 | kashyap | efried: Tack! | |
| 13:22:42 | kashyap | (Swedish for "thanks", if people don't want to look it up :D) | |
| 13:25:10 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add functional regression test for bug 1669054 https://review.openstack.org/649319 | |
| 13:25:11 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/649320 | |
| 13:25:11 | openstack | bug 1669054 in OpenStack Compute (nova) "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem) | |
| 13:36:40 | jaypipes | mnaser: around? can you please execute both of the queries in this pastebin and show me the output please? http://paste.openstack.org/show/748718/ | |
| 13:37:28 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Add functional regression test for bug 1669054 https://review.openstack.org/649325 | |
| 13:37:29 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/649326 | |
| 13:37:29 | openstack | bug 1669054 in OpenStack Compute (nova) stein "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem) | |
| 13:49:18 | yonglihe | mriedem: alex_xu: patch splited, and new unit test added. and... i need fix 2 of them. https://review.openstack.org/#/c/627765/13. thanks. | |
| 13:51:38 | alex_xu | yonglihe: sorry for not reach it for a while, will try tomorrow | |
| 13:52:07 | yonglihe | thanks, great news for me. | |
| 13:53:03 | efried | kashyap: I actually knew that :) | |
| 13:53:31 | kashyap | efried: Nice. Then I can try a few more phrases later on, then :D | |
| 13:54:23 | efried | In Swedish? That would be pretty much the extent of it for me. Hit me with some other languages though | |
| 13:55:06 | efried | kashyap: full disclosure, I wasn't sure of the spelling. The same pronunciation has the same meaning (but slightly different spellings) in Swedish, Danish, and Norwegian. | |
| 13:55:50 | kashyap | efried: Yeah, you're right -- they're all North Germanic languages | |
| 13:56:14 | kashyap | So to put the spelling thing to rest: Tack (Swedish), Takk (Norwegian), and Tak (Danish) :D | |
| 13:57:28 | ygk_12345 | hi all | |
| 13:58:14 | ygk_12345 | i having issues with spinning up vms on a compute node. they are forever in the scheduling state | |
| 13:58:37 | ygk_12345 | it is rocky setup OSA | |
| 14:04:23 | kashyap | ygk_12345: Hi, as noted in PM, I think you might get better responses on the more generic #openstack channel, where more admins / operators might hang out. | |
| 14:05:11 | efried | Something happens every day around 1am that makes sqla race like hell. | |
| 14:05:11 | efried | (e.g. http://logs.openstack.org/50/638150/7/check/openstack-tox-py27/345d1d1/testr_results.html.gz) | |
| 14:05:11 | efried | mriedem: Here's an interesting one: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Background%20on%20this%20error%20at%3A%20http%3A%2F%2Fsqlalche.me%2Fe%2Frvf5%5C%22 | |
| 14:12:38 | artom | sean-k-mooney, hah, midair collision :) In any case, thanks for the sanity check | |
| 14:12:40 | artom | sean-k-mooney++ | |
| 14:12:48 | artom | Doh, wrong channel | |
| 14:15:12 | ygk_12345 | any nova expert here ? | |
| 14:15:25 | ygk_12345 | tried in #openstack channel but no one could help me | |
| 14:17:37 | efried | ygk_12345: Have you had a look in the logs yet? | |
| 14:17:55 | ygk_12345 | efried: yes | |
| 14:18:20 | ygk_12345 | efried: it doesn't seem to indicate that the build started | |
| 14:18:22 | ygk_12345 | efried: http://paste.openstack.org/show/748717/ | |
| 14:18:37 | ygk_12345 | efried: it is in the scheduling state forever | |
| 14:20:35 | efried | ygk_12345: What about the n-sch and/or n-cpu logs? Anything appear to be hanging/repeating in there? | |
| 14:20:56 | ygk_12345 | efried: nothing this is the log I got so far | |
| 14:21:59 | ygk_12345 | efried: I see this "claim resources in the placement API for instance c2093503-d2d0-4401-8956-6a68a7d6e0dc claim_resources /openstack/venvs/nova-18.1.5/lib/python2.7/site-packages/nova/scheduler/utils.py:934" | |
| 14:22:00 | openstackgerrit | Chris Dent proposed openstack/nova master: Don't report 'exiting' when mapping cells https://review.openstack.org/649340 | |
| 14:25:14 | efried | ygk_12345: that was in the n-sch log presumably. Nothing after that? | |
| 14:25:16 | cdent | ygk_12345: did you discover hosts in your cells? | |
| 14:25:31 | ygk_12345 | cdent: efried yes | |
| 14:25:52 | ygk_12345 | cdent: how to validate it ? | |
| 14:25:59 | ygk_12345 | cdent: regarding cells ? | |
| 14:27:28 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 14:27:35 | cdent | listing your hypervisors | |
| 14:27:45 | cdent | I think you need to look at your n-cpu logs | |
| 14:27:48 | ygk_12345 | cdent: all the hypervisors are enabled | |
| 14:27:54 | mriedem | efried: i think that's this http://status.openstack.org/elastic-recheck/#1793364 | |
| 14:28:06 | ygk_12345 | cdent: where to look at the n-cpu logs ? | |
| 14:28:54 | cdent | ygk_12345: it depends on how you installed your openstack, but somewhere you have a nova-compute process, on the host which is or manage your hypervisor | |
| 14:29:41 | ygk_12345 | cdent: i see only nova-compute log in the hypervisor. Also I cant find any entry with the vm uuid in the nova-compute log | |
| 14:29:59 | cdent | nova-compute log and n-cpu log are the same thing | |
| 14:30:28 | cdent | the name depends on how things were installed | |
| 14:30:39 | ygk_12345 | cdent: ok then I dont find any vm uuid entries there and all seems to be fine on the hypervisor | |
| 14:32:11 | cdent | ygk_12345: then either in the conductor log or the scheduler log there should be some kind of error, after the timestamp of the "claim resources" | |
| 14:32:35 | cdent | grepping for the vm uuid won't be sufficient, you need to look through the log for an error or warning | |
| 14:33:28 | openstackgerrit | Hamdy Khader proposed openstack/nova master: Fix port update of host_id in case of baremetal instance https://review.openstack.org/649345 | |
| 14:35:38 | ygk_12345 | cdent: I find this in conductor log "Setting instance to ERROR state.: MessagingTimeout: Timed out waiting for a reply to message ID ef508c01e69841ae9f84356b7463165c" | |
| 14:35:56 | ygk_12345 | cdent: Failed to compute_task_build_instances: Timed out waiting for a reply to message ID ef508c01e69841ae9f84356b7463165c: MessagingTimeout: Timed out waiting for a reply to message ID ef508c01e69841ae9f84356b7463165c | |
| 14:36:31 | sean-k-mooney | ygk_12345: well that is the down call to the comptue to spawn the instance | |
| 14:36:41 | sean-k-mooney | i think | |
| 14:36:58 | cdent | ygk_12345: okay that's progress (in the sense of useful info). either your messaging bus (rabbitmq) is too busy or your conductor and compute host are unable to talk to one another over that bus | |
| 14:37:01 | sean-k-mooney | so on the compute you should be able to grep for ef508c01e69841ae9f84356b7463165c | |
| 14:37:16 | cdent | this might mean the compute node is misconfigured | |
| 14:37:35 | ygk_12345 | cdent: sean-k-mooney let me check | |
| 14:37:38 | sean-k-mooney | cdent: as in listneing to the wrong exchange | |
| 14:37:50 | cdent | ygk_12345: I'm sorry but I've got to go, good luck with it | |
| 14:38:28 | ygk_12345 | sean-k-mooney: i dont find any log entry on the compute node with that message ID | |
| 14:39:14 | ygk_12345 | sean-k-mooney: do u suspect its the rabbitmq issue with compute node ? | |
| 14:39:30 | sean-k-mooney | that would seam like the next most likely option | |
| 14:39:38 | ygk_12345 | sean-k-mooney: how to check that ? | |
| 14:40:03 | ygk_12345 | sean-k-mooney: but I dont think we made any changes to rabbitmq | |
| 14:40:14 | sean-k-mooney | am first you need to check the amqp setting in your nova.conf on both the conduction and compute node and make sure they are the same | |
| 14:40:51 | sean-k-mooney | the next thing to do would be to look a the topic queue and see if you can see the pending message | |
| 14:41:53 | ygk_12345 | sean-k-mooney: i see that all the rabbit servers are pingable from the compute node | |
| 14:44:15 | mnaser | jaypipes: around right now, ill do the explain | |
| 14:44:33 | ygk_12345 | sean-k-mooney: any idea ? | |
| 14:46:05 | mnaser | jaypipes: http://paste.openstack.org/show/748727/ | |
| 14:46:54 | sean-k-mooney | ygk_12345: have you logged into rabbitmq to see if the messages are still in the excahge queues | |
| 14:47:14 | ygk_12345 | sean-k-mooney: i dont see any queues around when I did list_queues | |
| 14:47:31 | ygk_12345 | sean-k-mooney: there r three rabbit containers | |
| 14:48:17 | sean-k-mooney | well presuably you are using the clustering plugin so from any of them you should see the same view | |
| 14:48:31 | ygk_12345 | sean-k-mooney: yes haproxy | |
| 14:48:41 | sean-k-mooney | haproxy is not the same thing | |