| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-18 | |||
| 15:24:00 | mriedem | alaski: when you get a minute, why do we need the cell_info stuff in https://review.openstack.org/#/c/369634/ ? - doesn't that defeat the purpose of having to run discover_hosts, i.e. we want auto-registration of hosts into a cell so that people/tools don't have to run discover_hosts manually. i'm thinking in at least the single-cell case auto-register is pretty simple. | |
| 15:25:01 | cdent | jdandrea: do you want to stop sooner or less soon? | |
| 15:25:04 | jdandrea | markus_z I'm looking at https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L519 ... hmm, no timeout to speak of (that I can see). | |
| 15:25:15 | adreznec | Hey markus_z. It's been a while since I did that, but I'd be happy to help with anything I can remember | |
| 15:25:30 | jdandrea | cdent I don't want to stop sooner or less soon, neither. I am just trying to determine if there's a scheduling-specific timeout at all. | |
| 15:25:49 | openstackgerrit | Claudiu Belu proposed openstack/nova: Hyper-V PCI Passthrough https://review.openstack.org/420614 | |
| 15:26:30 | jdandrea | cdent Looking at this I suspect there ... isn't? (a scheduler-specific timeout) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L519 | |
| 15:27:46 | jdandrea | cdent That is, there's a 60 second overall timeout to boot a server. But is there anything that puts a time limit on nova-scheduler to come up with an answer to begin with? | |
| 15:28:05 | edleafe | jdandrea: nothing that I'm aware of | |
| 15:28:19 | jdandrea | edleafe excellent, ty! | |
| 15:28:25 | jdandrea | (and that matches up with what I'm finding as well) | |
| 15:28:35 | jdandrea | I just wanted to be sure I wasn't missing something. ;) | |
| 15:30:22 | markus_z | adreznec: thanks in advance. What do you remember? :) I was wondering which account I have to use. My own? A new one? | |
| 15:30:44 | adreznec | markus_z: So on the rtd side I went through the basic registration steps (set up my account, create project, etc) w/ sphinx docs and set the URL to point at the git.o.o for the repo. | |
| 15:31:39 | cdent | EmilienM: can you ping me please when that debug job comes through with info? | |
| 15:31:50 | adreznec | Then in the advanced settings I had to configure it to use a virtualenv for builds w/ the right requirements file | |
| 15:32:05 | jdandrea | markus_z, cdent ... unless there's an oslo *messaging* timeout? https://github.com/openstack/nova/blob/4e747092bcb015303efc2ab13da98ef5ce575ec8/nova/scheduler/rpcapi.py#L127 | |
| 15:32:42 | adreznec | I think that was about it actually, the infra hook took care of the rest (notifying for rebuild, etc) | |
| 15:32:52 | EmilienM | cdent: yes | |
| 15:32:56 | EmilienM | cdent: I will | |
| 15:34:14 | markus_z | adreznec: Did you use the email address of a "functional Notes ID" for the registration? Or simply your own? | |
| 15:34:29 | openstackgerrit | Vladik Romanovsky proposed openstack/nova: libvirt: set vlan tag for macvtap on SRIOV VFs https://review.openstack.org/421186 | |
| 15:36:45 | adreznec | markus_z: I originally just used my own, then later on added our CI email address as a maintainer to the docs projects | |
| 15:37:54 | markus_z | adreznec: Ah, ok, makes sense. I'll simply start so as well. Thanks! | |
| 15:38:26 | adreznec | markus_z: Np, if you have any issues I can help with let me know | |
| 15:42:00 | cdent | edleafe, bauzas, jaypipes: I'm going to be on PTO tomorrow and friday, so if an rp/placement update is desired if someone could do it that would be great. Otherwise I'll pick it back up next week. | |
| 15:42:39 | edleafe | cdent: roger that | |
| 15:47:18 | bauzas | cdent: ack | |
| 15:48:09 | openstack | bug 1646896 in OpenStack Compute (nova) "System hangs when using NFS storage backend with loopback mounts" [Low,Confirmed] https://launchpad.net/bugs/1646896 | |
| 15:48:09 | openstackgerrit | Mihail proposed openstack/nova: proposed fix for bug 1646896 - System hangs when using NFS storage backend with loopback mounts https://review.openstack.org/422041 | |
| 15:48:15 | EmilienM | cdent: wait wait wait | |
| 15:48:23 | EmilienM | cdent: you're going on PTO? no please | |
| 15:48:25 | openstackgerrit | Vladik Romanovsky proposed openstack/nova: Adding vlans filed to Device tagging metadata https://review.openstack.org/405951 | |
| 15:48:34 | cdent | EmilienM: just for two days | |
| 15:48:51 | cdent | and I'm still here for another few hours today | |
| 15:49:03 | cdent | because you said please | |
| 15:49:03 | cdent | for you in particular | |
| 15:49:13 | bauzas | EmilienM, cdent: I missed today's discussion | |
| 15:49:22 | bauzas | anything I need to look at ? | |
| 15:49:31 | openstack | Launchpad bug 1657467 in tripleo "placement: unable to refresh compute resource provider record" [High,Triaged] | |
| 15:49:31 | EmilienM | bauzas: https://bugs.launchpad.net/tripleo/+bug/1657467 | |
| 15:49:45 | EmilienM | cdent: ♥ | |
| 15:49:55 | markus_z | aww | |
| 15:50:07 | markus_z | ;) | |
| 15:50:44 | bauzas | EmilienM: not sure I understand correctly, do we have the placement service running yet in TripleO? | |
| 15:51:25 | EmilienM | bauzas: yes | |
| 15:51:30 | kashyap | markus_z: Hey there | |
| 15:51:37 | EmilienM | bauzas: but placement api doesn't start because compute can't register the resource | |
| 15:51:38 | markus_z | kashyap: hey ho | |
| 15:51:50 | EmilienM | bauzas: it worked 5 days ago and now it's failing to register | |
| 15:51:59 | kashyap | markus_z: So...Michele has pinged on email about this review (https://review.openstack.org/#/c/313568/ -- Plugin to setup libvirt/QEMU from tar releases) | |
| 15:52:11 | kashyap | markus_z: I know you gave some feedback in December | |
| 15:52:19 | bauzas | EmilienM: the line before litterally means something went wrong with keystone | |
| 15:52:35 | markus_z | kashyap: uh oh spaghettio, another proof that I'm a lazy ass | |
| 15:52:39 | EmilienM | bauzas: with cdent, we're trying to figure if keystone resources are correctly created, but I doubt this is the issue. Keystone resources for placement have been here for a while and they work | |
| 15:52:39 | bauzas | ie. keystone giving you back a 401 | |
| 15:52:43 | kashyap | markus_z: No, no, you're not! | |
| 15:52:50 | markus_z | ;) | |
| 15:52:57 | kashyap | markus_z: I'm actually burnt looking at it, as it's been dragged for over a year | |
| 15:53:00 | EmilienM | bauzas: that's what log says, not sure it's true. | |
| 15:53:10 | kashyap | (It is spun off from a different review.) | |
| 15:53:17 | EmilienM | bauzas: the keystone resources are ready *after* compute service starts | |
| 15:53:30 | kashyap | markus_z: I think Michele should engage in IRC here if he really wants it in, and solicit feedback, and answer questions in real-time | |
| 15:53:36 | EmilienM | bauzas: dansmith fixed the issue by adding a loop, it worked but now it fails to register in the loop | |
| 15:53:38 | bauzas | EmilienM: mmm, okay | |
| 15:53:46 | kashyap | Too much lag via email I feel. | |
| 15:53:58 | bauzas | EmilienM: yeah, exactly, we changed from a try-once into a recheck | |
| 15:54:10 | bauzas | okay, so it would fail once but works after that | |
| 15:54:26 | markus_z | kashyap: Want me to have a look at it? | |
| 15:55:18 | kashyap | markus_z: If you have time, sure. But I am willing to test it and see what falls through. Just spinning up a new DevStack env with this patch on my slow test machine next to me | |
| 15:55:40 | EmilienM | bauzas: yes, it should (and it did a few days ago) work when puppet creates keystone resources for placement (service, endpoint, etc) | |
| 15:55:57 | EmilienM | bauzas: in a few min, I'll have output of keystone resources | |
| 15:56:05 | kashyap | markus_z: But if you have other priority things, feel free to ignore it. Because, if the submitter like to get this merged, they should engage more actively as well | |
| 15:56:11 | bauzas | EmilienM: what looks weird to me is that we don't have "Updated inventory" logs in http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_32_39_092 | |
| 15:56:14 | markus_z | kashyap: I put it on my todo list for tomorrow. | |
| 15:56:45 | markus_z | kashyap: TBH, having a test job for that is far too long on my todo list but I tabled it week after week. :-/ | |
| 15:56:49 | kashyap | Sure. markus_z - Just a heads-up: Tomorrow I'll only be working til 13 CET. Then I'll be AFK, travelling the whole day | |
| 15:56:57 | bauzas | EmilienM: just a brief explanation, we basically cache the list of RPs we create in a memory object to reduce the number of HTTP calls | |
| 15:57:13 | markus_z | kashyap: cool, no biggie | |
| 15:57:15 | kashyap | markus_z: You're not alone in feeling that guilt, I know it all too well | |
| 15:57:15 | bauzas | EmilienM: the log you mention is because we can't find the mapping in the cache for the RP | |
| 15:57:46 | bauzas | EmilienM: if the cache key corresponding to the compute is not found, that's probably because we failed updating the placement service, hence that log | |
| 15:58:04 | bauzas | EmilienM: if we would update that placement service, we put the result in the cach | |
| 15:58:06 | bauzas | cache | |
| 15:59:11 | EmilienM | bauzas: what could cause this mapping failing? | |
| 16:00:18 | bauzas | EmilienM: excellent question | |
| 16:00:25 | mriedem | melwitt: so on this auto-register thing for cells, i just remembered that nova-compute actually does already make calls to nova-api https://github.com/openstack/nova/blob/045f08ab8a57b90b00e4777ca71a20b5603c296c/nova/compute/manager.py#L6421 | |
| 16:00:27 | bauzas | EmilienM: that I'm trying to answer by looking at the logs | |
| 16:00:38 | mriedem | not even through rpc | |
| 16:01:21 | bauzas | EmilienM: mmmm http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_31_39_862 | |
| 16:01:46 | EmilienM | bauzas: I think this one is because the keystone resources are not created yet | |
| 16:02:02 | EmilienM | bauzas: let's see timestamps, gimme a sec | |
| 16:02:27 | EmilienM | http://logs.openstack.org/66/421366/1/check/gate-puppet-openstack-integration-4-scenario004-tempest-centos-7/f138bcc/logs/nova/nova-compute.txt.gz#_2017-01-17_17_31_39_862 | |
| 16:02:45 | EmilienM | so puppet managed keystone resources at 17h33 but nova compute tries to reach them at 17h31 | |
| 16:02:57 | melwitt | mriedem: looking ... compute/api will call the rpcapi next though | |
| 16:03:12 | EmilienM | bauzas: nova compute should retry later no? | |
| 16:03:32 | bauzas | EmilienM: I'm verifying that | |
| 16:03:48 | bauzas | in theory, we auto-heal every 60 secs | |
| 16:04:18 | EmilienM | even if we could say "tell puppet to create keystone resource BEFORE starting nova compute", this is odd and nova-compute should retry later | |
| 16:04:41 | melwitt | mriedem: I don't think that's calling nova-api though, is it? self.compute_api is an interface to making that rpc call, which will I guess just call the same nova-compute | |