| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-06 | |||
| 19:32:04 | mriedem | bauwser: i figured | |
| 19:32:07 | bauwser | just popping up here :) | |
| 19:32:08 | mriedem | tell them i'm sorry, | |
| 19:32:18 | mriedem | bauwser: however, if they care about how upstream stable releases are versioned, | |
| 19:32:25 | mriedem | there is a way they can get involved in monitoring that... | |
| 19:32:52 | bauwser | mriedem: no worries, I'll ask them to leave tar and feather at home | |
| 19:33:29 | openstackgerrit | Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775 | |
| 19:33:30 | openstackgerrit | Dan Smith proposed openstack/nova: Make the SingleCellSimple fixture a little more comprehensive https://review.openstack.org/410472 | |
| 19:33:42 | melwitt | superdan: I dunno if you saw the scrollback but I was thinking I wonder if the hosts api call is going out on the wrong topic ('compute' instead of 'cells' topic, since the rpc connects get cached via cell uuid) and that's why it doesn't get a response there. I just don't get how that would start up with your patch and not being failing as-is | |
| 19:33:47 | openstackgerrit | Matthew Edmonds proposed openstack/nova: update policy UT fixtures https://review.openstack.org/398610 | |
| 19:35:14 | superdan | melwitt: hmm | |
| 19:35:45 | superdan | melwitt: I just put up a test what just comments out the fanout to the cells, which will fail a bunch of things, but I want to see if it affects the host bit | |
| 19:36:04 | mriedem | bauwser: so where are we at with cleaning up https://review.openstack.org/#/c/392569/ ? | |
| 19:36:29 | leakypipes | superdan: I live in Florida. of course I do. | |
| 19:36:37 | melwitt | because what's failing is this 'intercell' communication, the 'cells' topic | |
| 19:37:02 | superdan | melwitt: well, it's not actually.. if you look, the cells-child is failing to call to the compute via compute topic, which is cascading up | |
| 19:38:28 | melwitt | superdan: I thought this is showing a send to the 'cells' topic? http://logs.openstack.org/75/396775/21/check/gate-tempest-dsvm-cells-ubuntu-xenial/2d615a2/logs/screen-n-api.txt.gz?#_2017-01-06_16_06_16_678 | |
| 19:38:40 | superdan | melwitt: right, but hang on | |
| 19:38:51 | melwitt | lemme look in the other logs again | |
| 19:38:56 | superdan | melwitt: http://logs.openstack.org/75/396775/21/check/gate-tempest-dsvm-cells-ubuntu-xenial/2d615a2/logs/screen-n-cell-child.txt.gz?level=TRACE#_2017-01-06_16_01_16_493 | |
| 19:39:08 | superdan | that's what gets called, which makes a call to compute, which times out | |
| 19:43:04 | melwitt | it says FANOUT topic 'cells.intercell.response' for the same req-id. you're saying the call to compute will go on the 'compute' topic though, and that's not responding and thus doesn't end up responding on the cells topic | |
| 19:43:29 | superdan | melwitt: but it's a call to compute_rpcapi | |
| 19:43:48 | superdan | File "/opt/stack/new/nova/nova/compute/rpcapi.py", line 920, in get_host_uptime | |
| 19:44:19 | melwitt | yeah. okay. so it goes on cells topic -> compute topic then replies go backward. hmm | |
| 19:44:29 | superdan | right | |
| 19:46:02 | cdent | leakypipes: you might find this amusing or interesting: https://tank.peermore.com/_/8760ec93-b39a-4dd5-849f-d880572fb2cd | |
| 19:46:39 | cdent | that's a gabbit using a new $HISTORY feature in gabbi + some yaml anchors to play games with the placement api | |
| 19:47:22 | cdent | ./stack.sh && openrc admin && gabbi-run http://localhost < that.yaml | |
| 19:47:44 | cdent | gonna flesh it out a bit more, but going to dinner now | |
| 19:48:18 | cdent | s/openrc/. openrc/ | |
| 19:50:00 | melwitt | superdan: for cells v1 do we have a host mapping set up that has the child cell transport_url in it? | |
| 19:50:27 | melwitt | sorry cell mapping | |
| 19:50:41 | superdan | melwitt: yeah, should, but hmm, I wonder if that points to the wrong thing | |
| 19:50:56 | superdan | melwitt: in the devstack job, do we really have any different mq exchanges or anything? | |
| 19:51:01 | melwitt | I noticed the nova.conf and nova-cells.conf have different transport urls. but I don't get how it ever worked then | |
| 19:51:09 | superdan | ooooh | |
| 19:51:10 | superdan | because | |
| 19:51:17 | superdan | I just changed that in a not yet merged dep patch :) | |
| 19:51:21 | superdan | check the depends-on | |
| 19:52:40 | melwitt | ahhh | |
| 19:53:22 | superdan | please oh please tell me you spot something wrong in there :D | |
| 19:53:26 | superdan | PUHLEASE | |
| 19:53:43 | superdan | otherwise leakypipes will have to shoot me, | |
| 19:53:50 | superdan | and I hear he has bad aim and might have to take a few shots | |
| 19:54:20 | melwitt | thinking ... I think we need two cells, one for the api and one for the compute, being that they have different transport urls? I dunno, still thinking | |
| 19:54:53 | superdan | no, because the cellsv1 messaging isn't switched I think | |
| 19:55:00 | superdan | but we do need the cell mappings we have to point to the right place, | |
| 19:55:13 | superdan | so that when something calls on compute, it does so on the right bus | |
| 19:55:39 | melwitt | right ... so is that get_transport_url getting the right thing, because it needs to get it out of the nova-cells.conf | |
| 19:55:41 | superdan | so, | |
| 19:55:58 | superdan | right, I was going to say, get_transport_url is probably the api cell url, | |
| 19:56:09 | superdan | not the compute-cell url, which is what we need right? | |
| 19:56:22 | mlavalle | leakypipes thanks for the help :-) | |
| 19:56:24 | melwitt | yeah, I'm hoping that's what's happening because otherwise I dunno | |
| 19:56:35 | melwitt | looking for what get_transport_url does | |
| 19:56:59 | superdan | yeah, | |
| 19:57:04 | superdan | I need "get_transport_url child_cell" | |
| 19:57:09 | superdan | sonofa | |
| 19:57:29 | melwitt | yeah, need to pass the virtual_host. I really hope that fixes things | |
| 19:58:21 | superdan | yeah | |
| 19:58:42 | openstackgerrit | Sylvain Bauza proposed openstack/nova: WIP/DNM : Put all the magic in a PlacementClient https://review.openstack.org/417111 | |
| 19:58:43 | openstackgerrit | Sylvain Bauza proposed openstack/nova: Expose a REST API for a specific list of RPs https://review.openstack.org/392569 | |
| 19:58:50 | openstackgerrit | Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775 | |
| 19:58:55 | superdan | melwitt: we shall see ^ | |
| 19:58:58 | superdan | melwitt: thanks for spotting that | |
| 19:59:17 | bauwser | mriedem: leakypipes: rebased https://review.openstack.org/392569 | |
| 19:59:22 | melwitt | fingers crossed | |
| 19:59:49 | bauwser | mriedem: leakypipes: and uploaded latest version of the placementclient module - but that's not a goal for now I think | |
| 20:07:12 | openstackgerrit | Ildiko Vancsa proposed openstack/nova: Implement new attach/detach Cinder flow https://review.openstack.org/330285 | |
| 20:07:13 | openstackgerrit | Ildiko Vancsa proposed openstack/nova: Remove check_attach https://review.openstack.org/335358 | |
| 20:07:57 | mriedem | ildikov: hey, btw, | |
| 20:08:10 | mriedem | that wip patch to use cinder v3 in nova - totally failed with the same endpoint not found that you were seeing | |
| 20:08:29 | mriedem | http://logs.openstack.org/82/385682/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/0a4185b/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-01-05_19_30_30_353 | |
| 20:08:50 | ildikov | mriedem: I'm glad to know it's not just my ghost | |
| 20:09:21 | ildikov | mriedem: I fixed it locally by adding the endpoint_template to the config, but I still experience weird things | |
| 20:09:59 | ildikov | mriedem: oh and tnx for the heads up on this :) | |
| 20:10:22 | openstackgerrit | Sarafraj Singh proposed openstack/nova: Add service_token for nova-cinder interaction https://review.openstack.org/397399 | |
| 20:11:09 | openstackgerrit | Sarafraj Singh proposed openstack/nova: Add service_token for nova-neutron interaction https://review.openstack.org/410394 | |
| 20:32:50 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 20:36:22 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 20:37:11 | melwitt | hm, grenade jobs failing with "/opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:199 : openstack server remove floating ip cinder_server1 172.24.5.6 | get() takes exactly 1 argument (2 given)" | |
| 20:48:30 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for hypervisors API https://review.openstack.org/417507 | |
| 20:57:13 | openstackgerrit | Claudiu Belu proposed openstack/nova: Adds Hyper-V OVS ViF driver https://review.openstack.org/140045 | |
| 21:10:31 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for hypervisors API https://review.openstack.org/417507 | |
| 21:36:38 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 21:43:34 | openstackgerrit | Claudiu Belu proposed openstack/nova: Hyper-V: fix live migration with CSVs https://review.openstack.org/380448 | |
| 21:53:34 | superdan | melwitt: well, that got it back down from insane-fail to lots-of-fail | |
| 21:53:38 | superdan | which I call major progress | |
| 21:54:11 | superdan | like, finally some sane failures: http://logs.openstack.org/75/396775/23/check/gate-tempest-dsvm-cells-ubuntu-xenial/c10b6b3/logs/screen-n-api.txt.gz?level=TRACE#_2017-01-06_20_19_05_133 | |
| 21:54:19 | melwitt | superdan: heh, yeah. I saw there's a lot of UnknownCell errors in there now but hopefully that won't be too hard to track down | |
| 21:55:05 | superdan | yeah, I dunno what that means, but.. it's something to chase at least | |
| 21:55:23 | melwitt | definitely a big improvement over crazy pants | |
| 21:55:33 | superdan | huh, it means instance.cell_name is not set | |
| 21:56:23 | melwitt | meaning instance wasn't assigned to a cell v1 cell somehow. or hasn't been scheduled yet | |
| 21:56:40 | superdan | yeah | |
| 21:59:39 | melwitt | hm, so when an instance record is written to a cell db, when it syncs to the top the cell_name is set based on the message routing path? | |
| 22:00:17 | melwitt | I don't think I ever knew how cell_name got set, but it looks like it happens in instance_update_at_top in cells/messaging | |
| 22:01:24 | superdan | that's..odd | |
| 22:01:31 | superdan | I guess so the cell doesn't have to know its name or something? | |