| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-13 | |||
| 14:33:35 | bauzas | EmilienM: so they aren't registered in the Nova DB, correct? | |
| 14:33:39 | EmilienM | bauzas: that is true | |
| 14:33:41 | bauzas | okay | |
| 14:33:43 | openstackgerrit | Vasyl Saienko proposed openstack/nova: WIP implement interface attach/detach in ironic virt driver https://review.openstack.org/419975 | |
| 14:33:56 | bauzas | EmilienM: in theory, that shouldn't be a problem | |
| 14:34:34 | EmilienM | bauzas: that was 2 weeks ago, wanna help? https://twitter.com/EmilienMacchi/status/814838163654975488 | |
| 14:34:54 | bauzas | heh | |
| 14:36:38 | bauzas | EmilienM: the error occurs on a fresh new install, not an upgrade ? | |
| 14:36:44 | openstackgerrit | Vladyslav Drok proposed openstack/nova: Generate necessary network metadata for portgroups https://review.openstack.org/388756 | |
| 14:36:47 | EmilienM | bauzas: fresh | |
| 14:36:56 | bauzas | ok | |
| 14:36:59 | EmilienM | bauzas: we haven't the CI job in place for upgrade yet (it's WIP) | |
| 14:37:08 | bauzas | ack | |
| 14:37:13 | EmilienM | actually, we have but it fails because of this cellv2 thing | |
| 14:37:18 | EmilienM | and mwhahaha is working on it | |
| 14:37:39 | EmilienM | https://review.openstack.org/#/q/topic:bug/1649341 | |
| 14:39:16 | bauzas | EmilienM: so, the goal of that simple_cells command is to populate your cellsv2 database with the necessary mappings for the compute nodes being in cells | |
| 14:39:37 | bauzas | EmilienM: if you don't have nodes already, then you have no cell mapping AFAICT | |
| 14:40:06 | bauzas | but when you add a node, then you have to run again the simple cells command in order to assign it a cell | |
| 14:40:17 | EmilienM | why isn't it dynamic? | |
| 14:40:30 | EmilienM | why conductor or something wouldn't do it for us? | |
| 14:40:35 | bauzas | because it hasn't been done yet | |
| 14:40:42 | EmilienM | ah ok, it's in the roadmap | |
| 14:40:46 | bauzas | well | |
| 14:40:49 | bauzas | the fact is | |
| 14:41:14 | bauzas | only the operator can know what cell the node should be added to, nova can't guess for it except putting the node into a default cell | |
| 14:41:59 | EmilienM | we could have the cell name in the nova.conf on each compute node | |
| 14:42:15 | EmilienM | and compute would send this name to conductor which write in db | |
| 14:42:36 | EmilienM | so you can scale up/down cells dynamically without running nova-manage | |
| 14:42:50 | bauzas | that's a possibility but like I said, it's not written yet :) | |
| 14:42:56 | EmilienM | cool np | |
| 14:43:00 | bauzas | so | |
| 14:43:16 | bauzas | I'm trying to understand where exactly puppet fails | |
| 14:43:26 | bauzas | when running the simple cell command | |
| 14:43:35 | bauzas | EmilienM: do you have some trace that I can look at ? | |
| 14:43:49 | bauzas | I guess only https://bugs.launchpad.net/nova/+bug/1656276 ? | |
| 14:43:49 | openstack | Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged] | |
| 14:43:50 | EmilienM | it's running "nova-manage cell_v2 simple_cell_setup" as part of the nova database initialization when no compute nodes have been created and it returns 1 | |
| 14:44:05 | openstackgerrit | Vladyslav Drok proposed openstack/nova: Support Ironic interface attach/detach in nova virt https://review.openstack.org/364413 | |
| 14:44:10 | EmilienM | yes there is no trace but I can easily give one to you, let me do something quickly | |
| 14:44:40 | openstackgerrit | Emilien Macchi proposed openstack/nova: Debug https://bugs.launchpad.net/tripleo/+bug/1656276 https://review.openstack.org/419926 | |
| 14:45:25 | EmilienM | bauzas: i'm preparing a CI job with more debug, let me 5 min | |
| 14:45:50 | bauzas | EmilienM: anyway, I think the bug description has a first point to look | |
| 14:46:26 | bauzas | EmilienM: so https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_cell_v2.pp#L36 means that we try 10 times every 5 secs until the return code is zero, right?N | |
| 14:46:40 | EmilienM | bauzas: "No hosts found to map to cell, exiting" | |
| 14:46:46 | EmilienM | bauzas: exactly | |
| 14:46:59 | EmilienM | bauzas: keep in mind we can change puppet to accept any return code for this Exec | |
| 14:47:08 | EmilienM | bauzas: but I'm afraid accepting 0,1 is not the right thing to do | |
| 14:47:12 | bauzas | EmilienM: okay, so why does the puppet command failed only once ? | |
| 14:47:22 | openstackgerrit | Vladyslav Drok proposed openstack/nova: Generate necessary network metadata for portgroups https://review.openstack.org/388756 | |
| 14:47:22 | EmilienM | because what is the exec fails for a good reason, we won't see it | |
| 14:47:25 | bauzas | EmilienM: it should have failed 10 times ? | |
| 14:47:45 | EmilienM | bauzas: it tried 10 times and show output of last failure | |
| 14:47:52 | EmilienM | but I guess it's 10 times the same failure | |
| 14:48:07 | bauzas | ah ok | |
| 14:48:08 | EmilienM | it returned 1 ten times | |
| 14:48:28 | EmilienM | this "No hosts found to map to cell, exiting" means "no compute has been found, fail"? | |
| 14:49:49 | bauzas | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1251 is the reason behind it | |
| 14:50:22 | bauzas | EmilienM: which returns 1 indeed | |
| 14:50:23 | bauzas | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1114 | |
| 14:50:42 | EmilienM | indeed | |
| 14:50:47 | amoralej | bauzas, EmilienM, i've been doing some additional tests and if no computes are found when running the command the cell is not added to cell_mappints table | |
| 14:51:03 | amoralej | so, it may be right to fail if no compute nodes are found | |
| 14:51:05 | bauzas | amoralej: that's an expected behaviour | |
| 14:51:22 | bauzas | so, the point is, what is the meaning of that return code equal to 1 | |
| 14:51:25 | EmilienM | so we need to run nova-compute *before* running this command? it might be very tricky | |
| 14:51:28 | openstackgerrit | John Garbutt proposed openstack/nova: WIP: Call preallocate_for_instance before scheduler https://review.openstack.org/413124 | |
| 14:51:32 | EmilienM | in multinode deployments specially | |
| 14:51:51 | amoralej | bauzas, then in any case we need to get compute nodes before running it | |
| 14:52:01 | bauzas | EmilienM: well, I think no computes nodes doesn't mean that the command failed | |
| 14:52:15 | EmilienM | bauzas: https://github.com/openstack/nova/commit/a86d78f5f048e11ddb7e375f2d6cc707108b9478 | |
| 14:52:15 | bauzas | EmilienM: so, there are 2 possibilites | |
| 14:52:27 | EmilienM | alaski: ^ | |
| 14:52:46 | bauzas | either we say it's okay to have no computes when running that command, and then we should soft-fail with return code 0 | |
| 14:53:08 | bauzas | or, we ask users to deploy compute nodes before running that command | |
| 14:53:29 | amoralej | bauzas, in option 1, we'd need to re-run it after adding the compute nodes? | |
| 14:53:37 | bauzas | amoralej: indeed | |
| 14:54:11 | amoralej | then, option 1 converts to option 2 | |
| 14:54:30 | amoralej | even if command exit with rc 0, we need to orchestrate | |
| 14:54:37 | bauzas | you're right | |
| 14:54:40 | mdbooth | lyarwood: That's a separate issue, right? | |
| 14:54:51 | mdbooth | I mean, isn't that broken either way? | |
| 14:54:58 | amoralej | bauzas, i have a doubt about cell0 | |
| 14:55:31 | bauzas | amoralej: EmilienM: I was a bit off the recent cellsv2 discussions, so I don't know if we're planning something like an automatic mapping when adding nodes | |
| 14:55:32 | amoralej | what i find when i'm running the command after adding compute nodes is that a new cell is created in cell_mappings | |
| 14:55:44 | bauzas | amoralej: cell0 is a total different story | |
| 14:55:52 | bauzas | amoralej: cell0 is not a cell :p | |
| 14:55:58 | bauzas | well, it is a cell | |
| 14:56:10 | bauzas | but we could have named it "cemetery cell" | |
| 14:56:28 | amoralej | ok, then i have a new doubt, :) | |
| 14:56:29 | bauzas | it's for failed instances not having a compute node | |
| 14:56:50 | bauzas | ie. when the scheduler returns NoValidHosts, then the instance is mapped to cell0 | |
| 14:56:59 | lyarwood | mdbooth: nope, my change introduces the lookup, previously all the calls made by BlockDevice to libvirt would work with either a path or target dev | |
| 14:57:03 | amoralej | so, for single cell deployments, do i need only cell0 or cell0 and "reall cell" ? | |
| 14:57:05 | bauzas | instead of being mapped to the cell that the compute belongs to | |
| 14:57:17 | mdbooth | lyarwood: Of course. Got it. | |
| 14:57:37 | bauzas | amoralej: for real deployments, you need cell0 *and* a single cell | |
| 14:57:52 | amoralej | ok, understood | |
| 14:58:19 | bauzas | amoralej: that's why we populate cell0 with a db sync | |
| 14:58:35 | bauzas | so that we have everything like if it was a normal cell | |