| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-02 | |||
| 19:37:40 | fried_rice | leakypipes: cool beans. | |
| 19:38:16 | fried_rice | sean-k-mooney, cfriesen, mriedem: I didn't dig too deep, but the tempest-full-py3 placement log on that trust-the-cache change is 154K, vs random other change where it's 305K. | |
| 19:38:47 | fried_rice | leakypipes: --^ (see also ML) | |
| 19:38:49 | mriedem | meaning we just hit the placement API less | |
| 19:38:56 | fried_rice | exactly, which was the point. | |
| 19:39:10 | mriedem | funny how removing a cache means we hit the API *less* | |
| 19:39:16 | fried_rice | No, we didn't remove the cache | |
| 19:39:22 | fried_rice | we removed *refreshing* the cache | |
| 19:39:25 | fried_rice | oh | |
| 19:39:54 | fried_rice | Actually the part that we removed by default was just the updating of the compute rp every minute. | |
| 19:39:55 | mriedem | those sound like married cousins | |
| 19:40:18 | fried_rice | to get the full effect of this thing, I would need to set that config var. | |
| 19:40:24 | fried_rice | does that happen in the devstack project? | |
| 19:40:33 | fried_rice | cause yeah, I totally want to do that. | |
| 19:40:45 | mriedem | it doesn't | |
| 19:40:52 | mriedem | you could post-config that in the nova-next job or something if you wanted | |
| 19:40:55 | mriedem | to see a run with it | |
| 19:41:06 | mriedem | [[post-config|$NOVA_CPU_CONF]] something like that | |
| 19:41:10 | mriedem | there are examples | |
| 19:41:26 | mriedem | you want to blast it up to 1000000 or something? | |
| 19:41:31 | fried_rice | mriedem: No, zero | |
| 19:41:39 | mriedem | i thought the min was 1? | |
| 19:41:41 | mriedem | or are you changing that? | |
| 19:41:42 | fried_rice | the patch lets you do that now. | |
| 19:41:45 | mriedem | ok | |
| 19:41:58 | fried_rice | zero => float('inf') kind of thing | |
| 19:42:01 | mriedem | then you could just put a patch on top that makes the nova-next job disable the refresh | |
| 19:42:30 | fried_rice | Okay, I have the bat and the blindfold. Can you spin me around three times and point me in the right direction? | |
| 19:42:51 | fried_rice | (I don't know where to find that config for the nova-next job) | |
| 19:43:15 | mriedem | it would go in here https://github.com/openstack/nova/blob/master/playbooks/legacy/nova-next/run.yaml#L36 | |
| 19:43:48 | fried_rice | ack, thx. I'm going to make a couple of edits per review comments from sean-k-mooney and then I'll sling that on top. | |
| 19:43:50 | mriedem | but in a [[post-config|$NOVA_CPU_CONF]] section | |
| 19:44:14 | mriedem | i think that will do the trick but it's been awhile since i've dug into my bag of tricks | |
| 19:44:21 | mriedem | 2 days to be exact | |
| 19:45:37 | openstackgerrit | Jim Rollenhagen proposed openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | |
| 20:04:58 | nicolasbock | Hi. I keep getting `nova-spicehtml5proxy: 2018-11-02 20:00:53.205 29786 ERROR nova.console.websocketproxy error: [Errno 32] Broken pipe` errors | |
| 20:05:10 | nicolasbock | Did I misconfigure something? | |
| 20:05:54 | melwitt | nicolasbock: what release are you on? did you upgrade recently? do you have multiple cells? | |
| 20:06:12 | mgagne | Do you still need to define [database]/connection in API cell or is [api_database]/connection enough? | |
| 20:09:32 | nicolasbock | melwitt (IRC): sorry I should have had this information ready before I asked... | |
| 20:13:10 | mriedem | mgagne: our CI points [database]/connection at the cell0 database for controller services and such | |
| 20:13:20 | mriedem | it won't have instances or hosts or anything, but we do register the services | |
| 20:14:17 | mgagne | mriedem: ok, how does it relate to the database_connection used by nova-manage cell_v2 map_cell0 ? isn't the config read once and never used after? | |
| 20:15:08 | mriedem | that goes into the cell_mappings record in the api db, | |
| 20:15:28 | mriedem | that's not what is used to create the services table entry though for the controller services like nova-osapi_compute, nova-conductor, and nova-scheduler | |
| 20:15:39 | mriedem | those go into the [database]/connection | |
| 20:16:28 | mgagne | mriedem: hooo, you mean when the service starts up and register itself | |
| 20:16:45 | mgagne | ok, it makes a lot of sense | |
| 20:18:30 | mriedem | yeah | |
| 20:18:42 | mriedem | nova.service.Service.start() or whatever | |
| 20:18:51 | mriedem | does an update-or-create dance with itself | |
| 20:18:59 | mriedem | like billy idol | |
| 20:19:37 | mgagne | :D | |
| 20:22:17 | nicolasbock | melwitt (IRC): nova-18.0.0.0b4.dev134, no upgrade, fresh install (OpenStack Ansible / master), cell0 and cell1 | |
| 20:24:30 | melwitt | nicolasbock: ok, thanks. and when do you get that error? when starting the proxy or when trying to get a new console or? | |
| 20:24:51 | melwitt | do you have a traceback you could paste? | |
| 20:24:56 | nicolasbock | it seems to show up after I create a server | |
| 20:25:00 | nicolasbock | Yes | |
| 20:25:08 | cdent | fried_rice: sanity check me on some mental meanderings please | |
| 20:26:05 | cdent | "generation" can act as a cache invalidator for a resource provider. you can compare generations in various ways, and if it is different from what you think it should be be, then you know to do differently | |
| 20:26:17 | nicolasbock | melwitt (IRC): http://paste.openstack.org/show/734053/ | |
| 20:26:41 | cdent | what if: when the generation on an rp changes, it caused a generation bump on its parent | |
| 20:27:09 | cdent | thus: whenever a compute node needing to validate it's tree, it just needed to compare stored generation with the generation of just the root | |
| 20:27:27 | cdent | s/needing/needed/ s/it's/its/ | |
| 20:27:59 | cdent | I'm thinking perhaps that's how nested generations should always have worked | |
| 20:31:58 | mriedem | fried_rice: thanks for the ML post, i dumped my brain | |
| 20:32:57 | melwitt | nicolasbock: hm, I haven't seen that before. where have you deployed the console proxy? is it on a host that has a nova.conf with [database]connection set to the cell1 database? | |
| 20:33:57 | melwitt | the traceback looks unrelated to database access issues, but asking just in case | |
| 20:35:26 | nicolasbock | melwitt (IRC): The console proxy is deployed in an LXC container that runs the Nova API | |
| 20:35:42 | nicolasbock | I haven't noticed database acces issues from that container to the Galera container | |
| 20:38:41 | melwitt | nicolasbock: ok. it sounds like it's possible the [spice] config options have been misconfigured, like the address configs. seeing if I can find docs. have you been able to set this up properly before in the past and now it is broken? | |
| 20:39:17 | nicolasbock | No, this is the first time I am setting this up. | |
| 20:39:25 | nicolasbock | I could try an older branch | |
| 20:40:30 | melwitt | no, that's ok, just checking whether this is a regression in behavior you have seen before | |
| 20:42:55 | nicolasbock | I have set `novncproxy_base_url = https://44.81.0.110:6080/vnc_lite.html` in the nova-api container | |
| 20:43:04 | nicolasbock | However, that IP is the IP of the OSA host | |
| 20:43:13 | nicolasbock | Is that possible the issue? | |
| 20:43:48 | melwitt | yeah, I was about to suggest something like that based on what I'm reading in https://docs.openstack.org/nova/latest/admin/remote-console-access.html#spice-console | |
| 20:44:31 | fried_rice | cdent: So far the philosophy has been that the client isn't supposed to "compare" generations at all. | |
| 20:45:08 | fried_rice | They're supposed to be opaque, introspected only by placement itself. | |
| 20:45:29 | cdent | fried_rice: they are supposed to be meaningless, but comparing mine to yours doesn't imply meaning | |
| 20:45:36 | nicolasbock | melwitt (IRC): The "controller" node is the one that runs the API? | |
| 20:46:08 | melwitt | nicolasbock: I notice you are using novnc config options but are running the spice proxy? | |
| 20:46:23 | fried_rice | cdent: It's a subtle distinction - testing for (in)equality versus gt/lt - but it's still treating it as non-opaque on the client. | |
| 20:46:26 | cdent | fried_rice: if we had done etags instead of generations, the way to accomplish the thing I'm describing is with a conditional GET. | |
| 20:46:53 | fried_rice | cdent: but putting that aside for the moment, what you're talking about is overloading the generation to do more than we're doing now. | |
| 20:46:57 | melwitt | nicolasbock: I think there it means the "controller" is where you are running the proxy. and the config options in the [spice] section are on the compute host (compute host is referring to where the proxy is running) | |
| 20:46:59 | nicolasbock | melwitt (IRC): the initial installation was configured to use the spice proxy, but as I was getting these error messages I reconfigured it | |
| 20:47:06 | cdent | fried_rice: I seem to recall seeing generation comparison in report client code, but I may be out of date | |
| 20:47:17 | cdent | no it is doing exactly how a generation is defined | |
| 20:47:20 | nicolasbock | melwitt (IRC): Thanks that clarifies this | |
| 20:48:20 | nicolasbock | melwitt (IRC): I should stop the spicehtml5proxy service | |
| 20:50:33 | melwitt | nicolasbock: I'm not sure. it doesn't look like the spice options are related to the proxy service | |
| 20:50:52 | melwitt | meaning, I think they are consumed by nova-compute, not the proxy | |
| 20:52:21 | melwitt | so I'd fix the options on the host running nova-compute first and then see what happens | |
| 20:54:21 | nicolasbock | Ok, thanks melwitt (IRC) I'll try that | |
| 21:37:18 | mgagne | mriedem: does it *need* to be cell0 database or could it be a different dedicated database? | |
| 21:40:15 | mriedem | mgagne: doesn't need to be cell0, | |
| 21:40:20 | mriedem | that's just what we use in devstack | |
| 21:40:25 | mriedem | b/c we have the api db, cell0 and cell1 | |