| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-06 | |||
| 19:35:45 | superdan | melwitt: I just put up a test what just comments out the fanout to the cells, which will fail a bunch of things, but I want to see if it affects the host bit | |
| 19:36:04 | mriedem | bauwser: so where are we at with cleaning up https://review.openstack.org/#/c/392569/ ? | |
| 19:36:29 | leakypipes | superdan: I live in Florida. of course I do. | |
| 19:36:37 | melwitt | because what's failing is this 'intercell' communication, the 'cells' topic | |
| 19:37:02 | superdan | melwitt: well, it's not actually.. if you look, the cells-child is failing to call to the compute via compute topic, which is cascading up | |
| 19:38:28 | melwitt | superdan: I thought this is showing a send to the 'cells' topic? http://logs.openstack.org/75/396775/21/check/gate-tempest-dsvm-cells-ubuntu-xenial/2d615a2/logs/screen-n-api.txt.gz?#_2017-01-06_16_06_16_678 | |
| 19:38:40 | superdan | melwitt: right, but hang on | |
| 19:38:51 | melwitt | lemme look in the other logs again | |
| 19:38:56 | superdan | melwitt: http://logs.openstack.org/75/396775/21/check/gate-tempest-dsvm-cells-ubuntu-xenial/2d615a2/logs/screen-n-cell-child.txt.gz?level=TRACE#_2017-01-06_16_01_16_493 | |
| 19:39:08 | superdan | that's what gets called, which makes a call to compute, which times out | |
| 19:43:04 | melwitt | it says FANOUT topic 'cells.intercell.response' for the same req-id. you're saying the call to compute will go on the 'compute' topic though, and that's not responding and thus doesn't end up responding on the cells topic | |
| 19:43:29 | superdan | melwitt: but it's a call to compute_rpcapi | |
| 19:43:48 | superdan | File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 920, in get_host_uptime | |
| 19:44:19 | melwitt | yeah. okay. so it goes on cells topic -> compute topic then replies go backward. hmm | |
| 19:44:29 | superdan | right | |
| 19:46:02 | cdent | leakypipes: you might find this amusing or interesting: https://tank.peermore.com/_/8760ec93-b39a-4dd5-849f-d880572fb2cd | |
| 19:46:39 | cdent | that's a gabbit using a new $HISTORY feature in gabbi + some yaml anchors to play games with the placement api | |
| 19:47:22 | cdent | ./stack.sh && openrc admin && gabbi-run http://localhost < that.yaml | |
| 19:47:44 | cdent | gonna flesh it out a bit more, but going to dinner now | |
| 19:48:18 | cdent | s/openrc/. openrc/ | |
| 19:50:00 | melwitt | superdan: for cells v1 do we have a host mapping set up that has the child cell transport_url in it? | |
| 19:50:27 | melwitt | sorry cell mapping | |
| 19:50:41 | superdan | melwitt: yeah, should, but hmm, I wonder if that points to the wrong thing | |
| 19:50:56 | superdan | melwitt: in the devstack job, do we really have any different mq exchanges or anything? | |
| 19:51:01 | melwitt | I noticed the nova.conf and nova-cells.conf have different transport urls. but I don't get how it ever worked then | |
| 19:51:09 | superdan | ooooh | |
| 19:51:10 | superdan | because | |
| 19:51:17 | superdan | I just changed that in a not yet merged dep patch :) | |
| 19:51:21 | superdan | check the depends-on | |
| 19:52:40 | melwitt | ahhh | |
| 19:53:22 | superdan | please oh please tell me you spot something wrong in there :D | |
| 19:53:26 | superdan | PUHLEASE | |
| 19:53:43 | superdan | otherwise leakypipes will have to shoot me, | |
| 19:53:50 | superdan | and I hear he has bad aim and might have to take a few shots | |
| 19:54:20 | melwitt | thinking ... I think we need two cells, one for the api and one for the compute, being that they have different transport urls? I dunno, still thinking | |
| 19:54:53 | superdan | no, because the cellsv1 messaging isn't switched I think | |
| 19:55:00 | superdan | but we do need the cell mappings we have to point to the right place, | |
| 19:55:13 | superdan | so that when something calls on compute, it does so on the right bus | |
| 19:55:39 | melwitt | right ... so is that get_transport_url getting the right thing, because it needs to get it out of the nova-cells.conf | |
| 19:55:41 | superdan | so, | |
| 19:55:58 | superdan | right, I was going to say, get_transport_url is probably the api cell url, | |
| 19:56:09 | superdan | not the compute-cell url, which is what we need right? | |
| 19:56:22 | mlavalle | leakypipes thanks for the help :-) | |
| 19:56:24 | melwitt | yeah, I'm hoping that's what's happening because otherwise I dunno | |
| 19:56:35 | melwitt | looking for what get_transport_url does | |
| 19:56:59 | superdan | yeah, | |
| 19:57:04 | superdan | I need "get_transport_url child_cell" | |
| 19:57:09 | superdan | sonofa | |
| 19:57:29 | melwitt | yeah, need to pass the virtual_host. I really hope that fixes things | |
| 19:58:21 | superdan | yeah | |
| 19:58:42 | openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP/DNM : Put all the magic in a PlacementClient https://review.openstack.org/417111 | |
| 19:58:43 | openstackgerrit | Sylvain Bauza proposed openstack/nova: Expose a REST API for a specific list of RPs https://review.openstack.org/392569 | |
| 19:58:50 | openstackgerrit | Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775 | |
| 19:58:55 | superdan | melwitt: we shall see ^ | |
| 19:58:58 | superdan | melwitt: thanks for spotting that | |
| 19:59:17 | bauwser | mriedem: leakypipes: rebased https://review.openstack.org/392569 | |
| 19:59:22 | melwitt | fingers crossed | |
| 19:59:49 | bauwser | mriedem: leakypipes: and uploaded latest version of the placementclient module - but that's not a goal for now I think | |
| 20:07:12 | openstackgerrit | Ildiko Vancsa proposed openstack/nova: Implement new attach/detach Cinder flow https://review.openstack.org/330285 | |
| 20:07:13 | openstackgerrit | Ildiko Vancsa proposed openstack/nova: Remove check_attach https://review.openstack.org/335358 | |
| 20:07:57 | mriedem | ildikov: hey, btw, | |
| 20:08:10 | mriedem | that wip patch to use cinder v3 in nova - totally failed with the same endpoint not found that you were seeing | |
| 20:08:29 | mriedem | http://logs.openstack.org/82/385682/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0a4185b/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-01-05_19_30_30_353 | |
| 20:08:50 | ildikov | mriedem: I'm glad to know it's not just my ghost | |
| 20:09:21 | ildikov | mriedem: I fixed it locally by adding the endpoint_template to the config, but I still experience weird things | |
| 20:09:59 | ildikov | mriedem: oh and tnx for the heads up on this :) | |
| 20:10:22 | openstackgerrit | Sarafraj Singh proposed openstack/nova: Add service_token for nova-cinder interaction https://review.openstack.org/397399 | |
| 20:11:09 | openstackgerrit | Sarafraj Singh proposed openstack/nova: Add service_token for nova-neutron interaction https://review.openstack.org/410394 | |
| 20:32:50 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 20:36:22 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 20:37:11 | melwitt | hm, grenade jobs failing with "/opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:199 : openstack server remove floating ip cinder_server1 172.24.5.6 | get() takes exactly 1 argument (2 given)" | |
| 20:48:30 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for hypervisors API https://review.openstack.org/417507 | |
| 20:57:13 | openstackgerrit | Claudiu Belu proposed openstack/nova: Adds Hyper-V OVS ViF driver https://review.openstack.org/140045 | |
| 21:10:31 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for hypervisors API https://review.openstack.org/417507 | |
| 21:36:38 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 21:43:34 | openstackgerrit | Claudiu Belu proposed openstack/nova: Hyper-V: fix live migration with CSVs https://review.openstack.org/380448 | |
| 21:53:34 | superdan | melwitt: well, that got it back down from insane-fail to lots-of-fail | |
| 21:53:38 | superdan | which I call major progress | |
| 21:54:11 | superdan | like, finally some sane failures: http://logs.openstack.org/75/396775/23/check/gate-tempest-dsvm-cells-ubuntu-xenial/c10b6b3/logs/screen-n-api.txt.gz?level=TRACE#_2017-01-06_20_19_05_133 | |
| 21:54:19 | melwitt | superdan: heh, yeah. I saw there's a lot of UnknownCell errors in there now but hopefully that won't be too hard to track down | |
| 21:55:05 | superdan | yeah, I dunno what that means, but.. it's something to chase at least | |
| 21:55:23 | melwitt | definitely a big improvement over crazy pants | |
| 21:55:33 | superdan | huh, it means instance.cell_name is not set | |
| 21:56:23 | melwitt | meaning instance wasn't assigned to a cell v1 cell somehow. or hasn't been scheduled yet | |
| 21:56:40 | superdan | yeah | |
| 21:59:39 | melwitt | hm, so when an instance record is written to a cell db, when it syncs to the top the cell_name is set based on the message routing path? | |
| 22:00:17 | melwitt | I don't think I ever knew how cell_name got set, but it looks like it happens in instance_update_at_top in cells/messaging | |
| 22:01:24 | superdan | that's..odd | |
| 22:01:31 | superdan | I guess so the cell doesn't have to know its name or something? | |
| 22:01:48 | melwitt | seems like it. I dunno | |
| 22:04:04 | superdan | man | |
| 22:04:07 | melwitt | superdan: I wonder if it's something like, it's reading the instance from the cell db so it doesn't have cell_name there. cell_name is only in the record at the top? | |
| 22:04:15 | superdan | I'm just shocked that this continues to work with none of us really knowing about it | |
| 22:04:25 | superdan | yeah might be | |
| 22:05:06 | melwitt | yeah well, we've taken care not to touch the code too much. and I broke things when I object-ified some stuff there that had to get fixed | |
| 22:07:42 | openstackgerrit | Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775 | |
| 22:07:44 | superdan | theory-testing hack ^ | |
| 22:07:51 | melwitt | I wonder if there are any other problems with reading instances from the compute cell db rather than the api cell db. I hope not | |
| 22:09:13 | superdan | well, | |
| 22:09:28 | superdan | it may surprise some people that don't expect connections from api to those dbs | |