| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-13 | |||
| 14:24:00 | openstackgerrit | garyk proposed openstack/nova: VMware: retry network lookup if network not ready https://review.openstack.org/391778 | |
| 14:25:26 | openstackgerrit | Vasyl Saienko proposed openstack/nova: Support Ironic interface attach/detach in nova virt https://review.openstack.org/364413 | |
| 14:30:45 | bauzas | EmilienM: hola, sorry was afk due to 10cm of snow and my frontyark needing cleanup :) | |
| 14:30:51 | bauzas | frontyard* even | |
| 14:30:58 | bauzas | EmilienM: owalsh pointed me the issue | |
| 14:31:03 | EmilienM | bauzas: 10 cm only? :-P | |
| 14:31:22 | bauzas | we're in France, and you know how 10cm of snow can be terrible for us :p | |
| 14:31:29 | bauzas | (even living in the French alps) | |
| 14:31:39 | bauzas | anyway | |
| 14:31:40 | EmilienM | bauzas: ok cool, let me know if there is something we need to change in the tripleo workflow or if you spot something in nova itself | |
| 14:31:52 | bauzas | so, owalsh pointed me the sequence that puppet does | |
| 14:32:13 | bauzas | https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_cell_v2.pp#L41 means that it runs after API DB sync, right? | |
| 14:32:23 | bauzas | EmilienM: ^ | |
| 14:32:24 | lyarwood | mdbooth: correction, we need them still thanks to _live_snapshot, I've updated the review with a comment. | |
| 14:32:35 | EmilienM | bauzas: right | |
| 14:32:53 | bauzas | EmilienM: okay, so you populated the DB, which is good | |
| 14:33:25 | bauzas | EmilienM: but you haven't started the compute nodes, right? | |
| 14:33:35 | bauzas | EmilienM: so they aren't registered in the Nova DB, correct? | |
| 14:33:39 | EmilienM | bauzas: that is true | |
| 14:33:41 | bauzas | okay | |
| 14:33:43 | openstackgerrit | Vasyl Saienko proposed openstack/nova: WIP implement interface attach/detach in ironic virt driver https://review.openstack.org/419975 | |
| 14:33:56 | bauzas | EmilienM: in theory, that shouldn't be a problem | |
| 14:34:34 | EmilienM | bauzas: that was 2 weeks ago, wanna help? https://twitter.com/EmilienMacchi/status/814838163654975488 | |
| 14:34:54 | bauzas | heh | |
| 14:36:38 | bauzas | EmilienM: the error occurs on a fresh new install, not an upgrade ? | |
| 14:36:44 | openstackgerrit | Vladyslav Drok proposed openstack/nova: Generate necessary network metadata for portgroups https://review.openstack.org/388756 | |
| 14:36:47 | EmilienM | bauzas: fresh | |
| 14:36:56 | bauzas | ok | |
| 14:36:59 | EmilienM | bauzas: we haven't the CI job in place for upgrade yet (it's WIP) | |
| 14:37:08 | bauzas | ack | |
| 14:37:13 | EmilienM | actually, we have but it fails because of this cellv2 thing | |
| 14:37:18 | EmilienM | and mwhahaha is working on it | |
| 14:37:39 | EmilienM | https://review.openstack.org/#/q/topic:bug/1649341 | |
| 14:39:16 | bauzas | EmilienM: so, the goal of that simple_cells command is to populate your cellsv2 database with the necessary mappings for the compute nodes being in cells | |
| 14:39:37 | bauzas | EmilienM: if you don't have nodes already, then you have no cell mapping AFAICT | |
| 14:40:06 | bauzas | but when you add a node, then you have to run again the simple cells command in order to assign it a cell | |
| 14:40:17 | EmilienM | why isn't it dynamic? | |
| 14:40:30 | EmilienM | why conductor or something wouldn't do it for us? | |
| 14:40:35 | bauzas | because it hasn't been done yet | |
| 14:40:42 | EmilienM | ah ok, it's in the roadmap | |
| 14:40:46 | bauzas | well | |
| 14:40:49 | bauzas | the fact is | |
| 14:41:14 | bauzas | only the operator can know what cell the node should be added to, nova can't guess for it except putting the node into a default cell | |
| 14:41:59 | EmilienM | we could have the cell name in the nova.conf on each compute node | |
| 14:42:15 | EmilienM | and compute would send this name to conductor which write in db | |
| 14:42:36 | EmilienM | so you can scale up/down cells dynamically without running nova-manage | |
| 14:42:50 | bauzas | that's a possibility but like I said, it's not written yet :) | |
| 14:42:56 | EmilienM | cool np | |
| 14:43:00 | bauzas | so | |
| 14:43:16 | bauzas | I'm trying to understand where exactly puppet fails | |
| 14:43:26 | bauzas | when running the simple cell command | |
| 14:43:35 | bauzas | EmilienM: do you have some trace that I can look at ? | |
| 14:43:49 | bauzas | I guess only https://bugs.launchpad.net/nova/+bug/1656276 ? | |
| 14:43:49 | openstack | Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged] | |
| 14:43:50 | EmilienM | it's running "nova-manage cell_v2 simple_cell_setup" as part of the nova database initialization when no compute nodes have been created and it returns 1 | |
| 14:44:05 | openstackgerrit | Vladyslav Drok proposed openstack/nova: Support Ironic interface attach/detach in nova virt https://review.openstack.org/364413 | |
| 14:44:10 | EmilienM | yes there is no trace but I can easily give one to you, let me do something quickly | |
| 14:44:40 | openstackgerrit | Emilien Macchi proposed openstack/nova: Debug https://bugs.launchpad.net/tripleo/+bug/1656276 https://review.openstack.org/419926 | |
| 14:45:25 | EmilienM | bauzas: i'm preparing a CI job with more debug, let me 5 min | |
| 14:45:50 | bauzas | EmilienM: anyway, I think the bug description has a first point to look | |
| 14:46:26 | bauzas | EmilienM: so https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_cell_v2.pp#L36 means that we try 10 times every 5 secs until the return code is zero, right?N | |
| 14:46:40 | EmilienM | bauzas: "No hosts found to map to cell, exiting" | |
| 14:46:46 | EmilienM | bauzas: exactly | |
| 14:46:59 | EmilienM | bauzas: keep in mind we can change puppet to accept any return code for this Exec | |
| 14:47:08 | EmilienM | bauzas: but I'm afraid accepting 0,1 is not the right thing to do | |
| 14:47:12 | bauzas | EmilienM: okay, so why does the puppet command failed only once ? | |
| 14:47:22 | openstackgerrit | Vladyslav Drok proposed openstack/nova: Generate necessary network metadata for portgroups https://review.openstack.org/388756 | |
| 14:47:22 | EmilienM | because what is the exec fails for a good reason, we won't see it | |
| 14:47:25 | bauzas | EmilienM: it should have failed 10 times ? | |
| 14:47:45 | EmilienM | bauzas: it tried 10 times and show output of last failure | |
| 14:47:52 | EmilienM | but I guess it's 10 times the same failure | |
| 14:48:07 | bauzas | ah ok | |
| 14:48:08 | EmilienM | it returned 1 ten times | |
| 14:48:28 | EmilienM | this "No hosts found to map to cell, exiting" means "no compute has been found, fail"? | |
| 14:49:49 | bauzas | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1251 is the reason behind it | |
| 14:50:22 | bauzas | EmilienM: which returns 1 indeed | |
| 14:50:23 | bauzas | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1114 | |
| 14:50:42 | EmilienM | indeed | |
| 14:50:47 | amoralej | bauzas, EmilienM, i've been doing some additional tests and if no computes are found when running the command the cell is not added to cell_mappints table | |
| 14:51:03 | amoralej | so, it may be right to fail if no compute nodes are found | |
| 14:51:05 | bauzas | amoralej: that's an expected behaviour | |
| 14:51:22 | bauzas | so, the point is, what is the meaning of that return code equal to 1 | |
| 14:51:25 | EmilienM | so we need to run nova-compute *before* running this command? it might be very tricky | |
| 14:51:28 | openstackgerrit | John Garbutt proposed openstack/nova: WIP: Call preallocate_for_instance before scheduler https://review.openstack.org/413124 | |
| 14:51:32 | EmilienM | in multinode deployments specially | |
| 14:51:51 | amoralej | bauzas, then in any case we need to get compute nodes before running it | |
| 14:52:01 | bauzas | EmilienM: well, I think no computes nodes doesn't mean that the command failed | |
| 14:52:15 | EmilienM | bauzas: https://github.com/openstack/nova/commit/a86d78f5f048e11ddb7e375f2d6cc707108b9478 | |
| 14:52:15 | bauzas | EmilienM: so, there are 2 possibilites | |
| 14:52:27 | EmilienM | alaski: ^ | |
| 14:52:46 | bauzas | either we say it's okay to have no computes when running that command, and then we should soft-fail with return code 0 | |
| 14:53:08 | bauzas | or, we ask users to deploy compute nodes before running that command | |
| 14:53:29 | amoralej | bauzas, in option 1, we'd need to re-run it after adding the compute nodes? | |
| 14:53:37 | bauzas | amoralej: indeed | |
| 14:54:11 | amoralej | then, option 1 converts to option 2 | |
| 14:54:30 | amoralej | even if command exit with rc 0, we need to orchestrate | |
| 14:54:37 | bauzas | you're right | |
| 14:54:40 | mdbooth | lyarwood: That's a separate issue, right? | |
| 14:54:51 | mdbooth | I mean, isn't that broken either way? | |
| 14:54:58 | amoralej | bauzas, i have a doubt about cell0 | |