| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-08 | |||
| 13:58:53 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Parse os attributes of LibvirtConfigGuest from xml https://review.openstack.org/410665 | |
| 13:58:53 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Parse filesystem elements of guest config https://review.openstack.org/410666 | |
| 13:58:54 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Use config types to parse XML for instance disks https://review.openstack.org/410667 | |
| 13:58:54 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Add missing tests for utils.find_disk https://review.openstack.org/414750 | |
| 13:58:55 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Parse basic attributes of LibvirtConfigGuest from xml https://review.openstack.org/414748 | |
| 13:58:56 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Use config types to parse XML for root disk https://review.openstack.org/411941 | |
| 13:58:56 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Introduce Guest.get_config method https://review.openstack.org/414749 | |
| 13:58:56 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Simplify internal usage of get_instance_disk_info https://review.openstack.org/334901 | |
| 13:58:57 | openstackgerrit | Feodor Tersin proposed openstack/nova master: libvirt: Remove unused disk_info parameter https://review.openstack.org/382025 | |
| 14:00:34 | mordred | sdague: thanks for reminding me I need to implement the needs-floating-ip flag | |
| 14:01:05 | mordred | sdague: it's one of the best examples, because discovering whether the cloud requires a floating ip or not involves _several_ api calls and a bunch of guess work | |
| 14:02:43 | sdague | mordred: yeh, so it feels like we should flip this whole thing over and start with the shade view of the world and how we'd effectively expose those things to you | |
| 14:05:02 | mordred | sdague: yah. I think that's likely a good and concrete starting point | |
| 14:05:45 | EmilienM | cdent: hey, I need your help if you have time | |
| 14:05:53 | mordred | sdague: re: putting stuff in the catalog, we have a cloud vendor profile data structure in os-client-config that might be an 'easy' thing to put into the catalog as a starting point | |
| 14:06:04 | cdent | EmilienM: I do, but I need to get some coffee first. cool? | |
| 14:06:08 | EmilienM | cdent: sure thing | |
| 14:06:16 | EmilienM | cdent: I'll paste things here | |
| 14:06:22 | mordred | (it needs a little clean up and more docs of course) | |
| 14:06:39 | EmilienM | so I'm deploying Nova from trunk, in eventlet (we removed WSGI/Apache for nova API like you suggested) | |
| 14:07:17 | EmilienM | we start nova services and then run the discover hosts command. Though it seems like the nova scheduler can't find the compute node registered | |
| 14:07:33 | EmilienM | and it looks like the compute didn't register to placement api: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_28_610 | |
| 14:07:40 | mordred | sdague: so like instead of this: http://git.openstack.org/cgit/openstack/os-client-config/tree/os_client_config/vendors/rackspace.json being in the os-client-config repo, it could just be in the catalog (which, of course, is mildly problematic because auth_url and auth_type still need to be discovered from somewhere, but baby steps) | |
| 14:08:26 | EmilienM | you can see nova-conductor which fails to find compute resource: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-conductor.txt.gz#_2017-02-08_06_41_56_037 | |
| 14:11:50 | johnthetubaguy | EmilienM: the nova-compute will need to be registered in its compute database before you run the discover_hosts cmd, it could be a bit racey, depending on when its called I guess? | |
| 14:12:28 | EmilienM | johnthetubaguy: so that's exactly what we were looking at | |
| 14:12:38 | EmilienM | johnthetubaguy: let me share timestamps with you | |
| 14:13:05 | EmilienM | at 06:38:25 we run nova-cell_v2-discover_hosts | |
| 14:13:21 | EmilienM | but we start nova compute at 06:32:06 | |
| 14:13:31 | johnthetubaguy | EmilienM: I think the three is being registered in the cell, registered in placement and registered in the API | |
| 14:13:46 | johnthetubaguy | let me find the compute log, one sec... | |
| 14:13:49 | EmilienM | we think nova compute is started at http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_34_19_476 | |
| 14:14:28 | johnthetubaguy | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_30_59_875 | |
| 14:14:56 | johnthetubaguy | thats where you see the compute node being created, i.e. registered in the child cell | |
| 14:15:01 | johnthetubaguy | you need to discover hosts after that | |
| 14:15:06 | johnthetubaguy | I think | |
| 14:16:14 | johnthetubaguy | hmm, I guess I don't understand the state of the system | |
| 14:16:28 | johnthetubaguy | it looks more like placement is not returning any hosts, I could be wrong | |
| 14:16:48 | owalsh | johnthetubaguy: I don't see any POST requests in the placement api log | |
| 14:17:07 | cdent | EmilienM, johnthetubaguy: I'm back, catching up. which is the best log to start with? | |
| 14:17:36 | johnthetubaguy | owalsh: yeah, compute logs say its not configured in the compute log, at least in the one I am looking at | |
| 14:17:53 | johnthetubaguy | owalsh: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_34_01_297 | |
| 14:18:20 | owalsh | johnthetubaguy: service isn't up yet | |
| 14:18:45 | johnthetubaguy | but thats a different log | |
| 14:19:06 | johnthetubaguy | I still see error here, looks like its configured by not able to reach it maybe | |
| 14:19:07 | johnthetubaguy | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_28_610 | |
| 14:19:25 | johnthetubaguy | owalsh: we look at the public endpoint in the service catalog for placement, in case that helps | |
| 14:19:35 | cdent | this is an error we were seeing a while back: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_34_19_311 | |
| 14:19:58 | johnthetubaguy | been looking at stuff with openstack-ansible folks last week, so wondering if its similar | |
| 14:23:22 | owalsh | johnthetubaguy: looks like it's created at 06:37:11 http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/keystone/keystone.txt.gz#_2017-02-08_06_37_11_641 | |
| 14:25:29 | owalsh | johnthetubaguy, cdent: and no more errors about the endpoint after this | |
| 14:25:37 | owalsh | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_38_22_609 | |
| 14:27:22 | sfinucan | sdague: nova-dhcpbridge only makes sense with nova-network, correct? https://linux.die.net/man/1/nova-dhcpbridge | |
| 14:31:12 | EmilienM | cdent, johnthetubaguy, owalsh: could it be something in nova.conf [placement] ? | |
| 14:31:35 | cdent | EmilienM: maybe | |
| 14:32:15 | EmilienM | cdent: the thing it, when we deploy nova from ocata-3, it works fine. Let me show you tripleo logs from our CI where nova trunk is not used | |
| 14:32:37 | johnthetubaguy | are those placement credentials passing the placement is_admin check, for example? | |
| 14:33:12 | EmilienM | cdent: http://logs.openstack.org/47/430447/1/check/gate-tripleo-ci-centos-7-nonha-multinode/dd43509/logs/subnode-2/var/log/nova/nova-placement-api.txt.gz#_2017-02-07_23_06_08_825 | |
| 14:33:15 | johnthetubaguy | EmilienM: thats probably just after we starting asking placement which host to use, potentially | |
| 14:33:24 | EmilienM | cdent: that is a CI job that doesn't run nova from trunk, but nova from ocata-3 | |
| 14:33:59 | EmilienM | johnthetubaguy: what is is_admin check ? | |
| 14:34:07 | cdent | EmilienM: can you enumerate what's different on the tripleo side, besides the switch to using nova-api with eventel (if anything)? | |
| 14:34:19 | mordred | sdague: something like this: https://review.openstack.org/430967 | |
| 14:34:19 | EmilienM | oh is_admin in keystone | |
| 14:34:30 | johnthetubaguy | EmilienM: placement logs didn't show any 403s so I am wrong, I was thinking is it an admin user | |
| 14:34:41 | Guest94046 | johnthetubaguy: yep, o3 won't break without placement | |
| 14:38:08 | EmilienM | cdent: what do you want to know exactly?? the diff between the 2 jobs? or the history of what we changed recently? | |
| 14:40:30 | cdent | EmilienM: I'm not sure. It would just be useful to be able to figure out what if anything may have impacted what's going on. | |
| 14:40:45 | EmilienM | the major thing is that we switched back to eventlet | |
| 14:42:58 | dansmith | EmilienM: not for n-cpu which is where this problem is, right? | |
| 14:44:03 | EmilienM | dansmith: I'm afraid I don't understand the question. We switched Nova API to not run in WSGI with apache, but run in classic eventlet. What are you asking about nova compute? | |
| 14:44:35 | dansmith | EmilienM: the problem is that nova-compute is not reporting into placement for the subnode right? | |
| 14:44:44 | johnthetubaguy | FWIW, this error log on the first compute node seems odd to me, not sure if thats normal: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_11_488 | |
| 14:44:48 | EmilienM | cdent: in the logs, I see "Placement service credentials do not work." 3 times but not more so I guess it could register at some point? | |
| 14:45:16 | mriedem | johnthetubaguy: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_11_488 is 'normal' | |
| 14:45:23 | mriedem | we should probably drop that down from an error | |
| 14:45:26 | EmilienM | dansmith: the subnode is a single node overcloud, consider it as a all-in-one node | |
| 14:45:40 | cdent | EmilienM: there's no evidence that the placement application has ever been contacted (in the web logs), unless I'm looking at the wrong ones | |
| 14:45:49 | mriedem | johnthetubaguy: https://github.com/openstack/nova/blob/d48aeb5783ea9bd9495eb2e7c67bc587dfd7d85d/nova/compute/manager.py#L6597 | |
| 14:45:56 | EmilienM | dansmith: node1 == undercloud, which is completely another story (and by the way, undercloud can deploy the overcloud fine in OVB jobs, that run Nova / Ironic) | |
| 14:45:57 | dansmith | the passwords in the two nodes for placement look very different | |
| 14:46:19 | EmilienM | dansmith: yeah don't look at subnode1 please | |
| 14:46:27 | dansmith | alright | |
| 14:46:29 | EmilienM | dansmith: that's another instance of nova, that we don't use in this case | |
| 14:46:42 | EmilienM | subnode2 is the only logs we should look today | |
| 14:46:57 | EmilienM | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/ | |
| 14:47:15 | johnthetubaguy | mriedem: OK, I clearly have less understanding of how this works than I thought | |
| 14:47:22 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Improve assertJsonEqual error reporting https://review.openstack.org/428199 | |
| 14:47:23 | dansmith | oh, I'm not looking at subnode1 | |
| 14:47:28 | dansmith | I'm looking at -2 | |
| 14:47:33 | EmilienM | ok good | |
| 14:47:43 | dansmith | or are you calling the primary node "subnode1" ? | |
| 14:48:21 | EmilienM | so: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/var/log/nova/ is the undercloud and please don't look at it we don't care now | |
| 14:48:36 | EmilienM | http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/ is overcloud, and we currently can't spawn a VM on it | |
| 14:48:49 | johnthetubaguy | according to placement, its only ever been reached by the scheduler I guess | |
| 14:48:53 | EmilienM | FYI I'm using https://review.openstack.org/#/c/430571/ for testing | |
| 14:49:26 | cdent | johnthetubaguy: yes (I was looking at the main logs when I saw no activity above, not the subnode-2, but subnode-2 is just hearing from the scheduler) | |
| 14:49:41 | johnthetubaguy | I just wish we had more details on why compute isn't able to talk to placement in here: http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_43_28_610 | |
| 14:50:20 | cdent | johnthetubaguy: shortly before that; http://logs.openstack.org/71/430571/1/check/gate-tripleo-ci-centos-7-nonha-multinode/bae4746/logs/subnode-2/var/log/nova/nova-compute.txt.gz#_2017-02-08_06_40_24_634 | |
| 14:50:33 | dansmith | ot | |