| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-27 | |||
| 12:55:25 | mriedem | nova-cpu.conf and nova_cell1.conf aren't used | |
| 12:55:47 | mriedem | nova-cpu.conf wouldn't even work b/c it doesn't have any information in it about which compute driver to use, or how to talk to placement/cinder/neutron/etc | |
| 12:56:35 | mriedem | http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/devstacklog.txt.gz#_2017-07-27_12_04_56_099 | |
| 12:56:35 | mriedem | the cell1 mapping is created here: | |
| 12:56:42 | mriedem | nova-manage --config-file /etc/nova/nova.conf --config-file /etc/nova/nova_cell1.conf cell_v2 create_cell --name cell1 | |
| 12:56:50 | mriedem | note that is using nova_cell1.conf rather than nova.conf | |
| 12:56:52 | sdague | mriedem: sure, where would the message queue get set | |
| 12:57:17 | sdague | maybe that's the missing piece, dumping the cell mappings | |
| 12:57:32 | mriedem | well we know the mq and db for cell1, it's taken from http://logs.openstack.org/58/487458/3/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/7aa067c/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 12:57:43 | sdague | right | |
| 12:57:50 | sdague | but that's not used anywhere | |
| 12:58:01 | mriedem | it's used when creating the cell1 mapping | |
| 12:58:02 | sdague | so that's stating that the cell1 mq is going to be on a vhost | |
| 12:58:06 | mriedem | nova-manage --config-file /etc/nova/nova.conf --config-file /etc/nova/nova_cell1.conf cell_v2 create_cell --name cell1 | |
| 12:58:13 | sdague | but nova-compute is started not listening to that vhost | |
| 12:59:32 | mriedem | right nova-compute is listening on rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/ | |
| 12:59:42 | mriedem | the cell1 mapping is sending to rabbit://stackrabbit:secretrabbit@10.16.80.100:5672/nova_cell1 | |
| 12:59:48 | sdague | but conductor isn't sending messages there, right? | |
| 13:00:09 | mriedem | because https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1042 | |
| 13:00:21 | mriedem | conductor does an mq switch when it casts to compute | |
| 13:00:21 | sdague | https://github.com/openstack-dev/devstack/blob/9596fdddccd04c26aa5adb923b9bd8e64c6593ec/lib/nova#L709-L712 | |
| 13:01:02 | sdague | mriedem: ok, so are you agreeing or disagreeing with me that nova-cond and nova-compute aren't talking on the same mq :) | |
| 13:01:37 | mriedem | can i phone a friend? | |
| 13:01:50 | mriedem | i think i'm agreeing, | |
| 13:01:57 | mriedem | and that would explain why we're not seeing the message get to n-cpu | |
| 13:02:26 | sdague | yeh | |
| 13:02:37 | sdague | is there a nova-manage command to dump the cell mappings? | |
| 13:02:46 | sdague | I think that's kind of critical to see the mismatch | |
| 13:03:48 | mriedem | yes, | |
| 13:03:52 | mriedem | nova-manage cell_v2 list_cells | |
| 13:04:17 | mriedem | with --verbose | |
| 13:04:28 | mriedem | --verbose dumps the db and mq urls | |
| 13:05:25 | mriedem | fwiw, this is a run before the fleetify patch where we create cell1 | |
| 13:05:26 | mriedem | http://logs.openstack.org/23/485823/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/206b79e/logs/devstacklog.txt.gz#_2017-07-21_01_52_17_794 | |
| 13:05:48 | mriedem | nova-manage cell_v2 create_cell --transport-url rabbit://stackrabbit:secretrabbit@10.0.1.31:5672/ --name cell1 | |
| 13:05:51 | mriedem | there is a single nova.conf | |
| 13:06:13 | mriedem | and it's using the same transport_url http://logs.openstack.org/23/485823/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/206b79e/logs/etc/nova/nova.conf.txt.gz | |
| 13:06:32 | mriedem | so yeah, i think that's the problem, the cell1 mapping is using the nova_cell1 mq and nova-compute is using the main mq | |
| 13:08:44 | sdague | ok, fix proposed | |
| 13:08:57 | sdague | including the dump of the cell mapping | |
| 13:10:50 | mriedem | i left a comment in ps2, | |
| 13:10:55 | mriedem | but it looks like you addressed it in ps4 | |
| 13:11:07 | mriedem | with https://review.openstack.org/#/c/487809/4/lib/nova@592 | |
| 13:14:19 | dansmith | mriedem: oh snap.. just woke up, but excellent call | |
| 13:15:09 | mriedem | sdague connected the dots for me, | |
| 13:15:12 | mriedem | plus sleep helps | |
| 13:15:25 | dansmith | I should have thought of that | |
| 13:15:39 | mriedem | hard to think of anything at the end of a day like yesterday | |
| 13:16:06 | sdague | mriedem: oh, sorry, I wasn't even looking at comments, I was just reading logs trying to understand | |
| 13:17:34 | mriedem | cdent: regarding functional testing, i was thinking the same yesterday, but didn't have time, | |
| 13:17:52 | vdrok | morning dansmith , missed all the fun :) | |
| 13:18:23 | dansmith | vdrok: me too, but.. good morning vdrok :P | |
| 13:18:29 | mriedem | but was thinking it could be relatively simple to write a functional test that starts 2 compute services, creates a server, checks allocations are just on the source host, does a resize to the 2nd compute, checks allocations are retained for both nodes, then does a resize confirm (and another test that does a resize revert), and then validates allocations after that is done | |
| 13:19:15 | cdent | relatively | |
| 13:19:34 | mriedem | cdent: but me working on something like that probably can't happen until tomorrow at this rate | |
| 13:19:51 | mriedem | i actually have *gasp* family commitments tonight | |
| 13:20:11 | cdent | I’m sicker than sick at the moment, but bored enough to still be hanging out | |
| 13:21:26 | sean-k-mooney | mriedem: hi i know your pretty busy with the ff today but any chance of inlcuding the 1 patch for mellanox's ovs offload and the 2 required for our nic feature based scheudling blueprint in pike or will they be pushed to queens? | |
| 13:22:08 | kashyap | mdbooth: When you get a moment, hope I addressed all of your remarks here: "libvirt: Post-migration, set cache value for Cinder volume(s)" -- https://review.openstack.org/#/c/485752/ | |
| 13:22:22 | gibi | mriedem: do we already have functional test that boot VMs which really uses the placement service. If there is such then I can try to put together the above suggested resize test | |
| 13:22:31 | mriedem | sean-k-mooney: i think the ovs offload from moshele is probably doable - i think i'd like a bp in nova for that though, just something simple to mirrow the neutron RFE | |
| 13:22:38 | mriedem | since nova doesn't do RFE bugs for blueprints | |
| 13:22:53 | mriedem | gibi: we do | |
| 13:23:09 | mriedem | gibi: the PlacementFixture is used in the _IntegratedHelpersMixin | |
| 13:23:13 | mriedem | which several functional tests use | |
| 13:23:26 | gibi | mriedem: cool, then I will put something together | |
| 13:23:35 | mriedem | there are other functional regression tests which don't use that mixin but still use the placement fixture to avoid warnings from the compute services in the logs | |
| 13:23:39 | gibi | mriedem: I anyhow wanted to test VM moving with placement | |
| 13:23:55 | mriedem | gibi: cool, that would be something that builds on https://review.openstack.org/#/c/483566/ | |
| 13:24:07 | mriedem | the other tricky thing though is the test has to use the filter scheduler, | |
| 13:24:08 | sean-k-mooney | mriedem: ok good to know. i saw that netronomes patchs seem to have merged so im sure jangutter is happy with that it would be nice to finish netronomes version too. im sure it will make moshele equally happy | |
| 13:24:13 | mriedem | lots of our functional tests just use the chance scheduler | |
| 13:24:22 | mriedem | gibi: so you'd have to use the filter scheduler with a minimal set of filters enabled | |
| 13:24:30 | mriedem | like RetryFilter and ComputeFilter i think | |
| 13:24:41 | gibi | mriedem: I'm aware of that based on the server group functional tests | |
| 13:24:58 | sdague | dansmith / mriedem is there a reason to not make --verbose the default behavior for list_cells | |
| 13:25:15 | sean-k-mooney | s/netronome/mellanox/ | |
| 13:25:18 | dansmith | sdague: makes it harder to be used by scripts | |
| 13:25:25 | mriedem | gibi: this is one that i worked on https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1671648.py | |
| 13:25:29 | mriedem | host most of the things yo'ud need i think | |
| 13:25:33 | sdague | I'm trying to understand the usage for just cellid | |
| 13:25:34 | dansmith | mriedem: sdague: is someone fixing the cell mapping? | |
| 13:25:34 | jangutter | sean-k-mooney, mriedem: yep, I'm pretty happy with moshele's patch -> it's cross-platform enough for us to use as well. | |
| 13:25:36 | mriedem | 2 nodes | |
| 13:25:39 | mriedem | filter scheduler + placement | |
| 13:25:41 | sdague | dansmith: in what way? | |
| 13:25:48 | mriedem | you'd just need to remove the part about it making one compute fail | |
| 13:25:52 | gibi | mriedem: cool, thanks for the pointer | |
| 13:26:04 | sdague | dansmith: doesn't my ps4 do that? | |
| 13:26:12 | dansmith | sdague: I haven't looked | |
| 13:26:21 | mriedem | dansmith: https://review.openstack.org/#/c/487809/4 | |
| 13:26:33 | mriedem | sdague: we don't use --verbose by default for list_cells since it dumps your transport and db mqs | |
| 13:26:37 | mriedem | s/mqs/urls/ | |
| 13:26:42 | mriedem | which have creds in them | |
| 13:26:55 | sdague | mriedem: ok, so run it through secret filter | |
| 13:27:14 | mriedem | do we have a secret filter? | |
| 13:27:16 | sdague | I guess, it's not really very discoverable that to get the good stuff you need --verbose | |
| 13:27:33 | mriedem | --dump-your-secrets? | |
| 13:27:43 | mriedem | --hand-over-your-cash? | |
| 13:28:01 | sdague | https://github.com/openstack/oslo.utils/blob/f69aa3c3866f1d1800184b6c905c77988278cd18/oslo_utils/strutils.py#L248 | |