| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-01 | |||
| 11:05:56 | johnthetubaguy | mdbooth: I was assuming a single attachment, but its a good question | |
| 11:06:47 | lyarwood | mdbooth: I would also say single, why? | |
| 11:07:02 | sfinucan | tonyb: Couldn't find logs/emails about that issue :/ Anywho, we're stuck with Sphinx 1.3.6 you mean? | |
| 11:07:38 | mdbooth | lyarwood: Just trying to think of problems which would be hard to fix | |
| 11:07:39 | tonyb | sfinucan: for a little longer | |
| 11:08:10 | sfinucan | tonyb: Darn. | |
| 11:08:14 | tonyb | sfinucan: http://lists.openstack.org/pipermail/openstack-dev/2017-March/113085.html | |
| 11:10:17 | sfinucan | tonyb: Just for context, why can't we use pbr (and where can't we use it) if it's been capped in some projects | |
| 11:10:55 | tonyb | sfinucan: that's cool it's a little complex | |
| 11:12:15 | tonyb | sfinucan: upper-constraints.txt os a list of every python library need by some part of OpenStack that we knwo works (at least installs) together | |
| 11:12:35 | tonyb | sfinucan: and all nice projects (such as nova) install *only taht version | |
| 11:13:12 | tonyb | sfinucan: sadly when we try to set the version of pbr === 2.0.0 in u-c.txt there are a small number of libraries that can't be installed | |
| 11:13:36 | tonyb | sfinucan: and therefore no part of OpenStack can use that version until they all agree that 2.0.0 is ok | |
| 11:14:20 | tonyb | sfinucan: upper-constraints.txt does a bunch of other things but it's beyond my ability to expalin on IRC late at night (for me) | |
| 11:14:57 | sfinucan | tonyb: ahhh, gotcha | |
| 11:15:56 | tonyb | sfinucan: there are reviews and PRs out to move the ball forward so we'll see how quickly they land and then we can take the next step | |
| 11:16:23 | sfinucan | tonyb: Perfect. Any topic I can look at? | |
| 11:16:31 | sfinucan | Throw a few reviews in | |
| 11:16:56 | tonyb | sfinucan: https://review.openstack.org/#/q/topic:bug/1668848 | |
| 11:17:38 | sfinucan | tonyb: fwiw, I'm pretty sure this has been said many times but you can't bump Sphinx until everything is using pbr 2.0, because 1.10 is broken with Sphinx 1.5 | |
| 11:17:47 | sfinucan | tonyb: Spot on. Thanks | |
| 11:18:45 | tonyb | sfinucan: right. | |
| 11:19:12 | sfinucan | tonyb: That also would suggest we should be bumping pbr to 2.0, perhaps? | |
| 11:19:17 | sfinucan | or at least blacklisting 1.10 | |
| 11:20:33 | tonyb | sfinucan: Yeah we'll set the minimum to 2.0.0 once if tested it a bit | |
| 11:21:20 | sfinucan | tonyb: Cool. Could we blacklist 1.10 also though? Allows us to start using Sphinx 1.5 | |
| 11:21:24 | sfinucan | (eventually) | |
| 11:21:37 | tonyb | sfinucan: Yeah we could | |
| 11:22:26 | sfinucan | OK. Guess that can be done later though | |
| 11:34:22 | kevinbenton | johnthetubaguy: how is your metadata-fu? | |
| 11:35:04 | johnthetubaguy | kevinbenton: rusty, but whats up? | |
| 11:35:18 | kevinbenton | johnthetubaguy: debugging a failure i observed in the gate | |
| 11:35:25 | kevinbenton | johnthetubaguy: one of the cirros VMs is not getting an SSH key | |
| 11:35:45 | johnthetubaguy | oh, interesting, almost like its fetching metadata before the ssh key is present in the metadata service | |
| 11:36:00 | johnthetubaguy | was it using config drive or not? | |
| 11:36:14 | kevinbenton | johnthetubaguy: no config drive | |
| 11:36:18 | kevinbenton | johnthetubaguy: just metadata | |
| 11:36:27 | kevinbenton | johnthetubaguy: i traced the request all of the way to nova API | |
| 11:36:33 | kevinbenton | johnthetubaguy: but the debug logs there aren't very helpful | |
| 11:36:40 | johnthetubaguy | ah, cool, you got a link to those | |
| 11:36:49 | kevinbenton | johnthetubaguy: http://logs.openstack.org/27/436427/2/gate/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/2f3e394/logs/screen-n-api.txt.gz | |
| 11:36:54 | johnthetubaguy | at least the request got there, that was my main question | |
| 11:37:02 | kevinbenton | johnthetubaguy: request-id in question is req-93182a9d-baa9-4cf5-8648-aa9c4a49500 | |
| 11:37:41 | kevinbenton | johnthetubaguy: whatever it is returning there is making cirros think there is nothing further so it doesn't continue on to query the openssh url for the actual keys | |
| 11:39:19 | kevinbenton | johnthetubaguy: a good request ID in the same tenant and behind the same router is req-ff83fad3-86cc-4097-aa6e-538c6889eae2 | |
| 11:39:42 | kevinbenton | johnthetubaguy: would digging up the corresponding instance IDs be helpful? | |
| 11:40:21 | johnthetubaguy | not sure that helps in the logs, I guess it would let you find the create instance requests etc, int he compute logs which would have a hint | |
| 11:40:33 | johnthetubaguy | kevinbenton: this is one of those cases where we just get ssh didn't work errors right? | |
| 11:40:49 | kevinbenton | johnthetubaguy: well in this case it's explicit authentication failures | |
| 11:40:59 | johnthetubaguy | ah, right, that makes sense | |
| 11:41:00 | kevinbenton | johnthetubaguy: 2017-02-28 11:48:29,144 12602 INFO [paramiko.transport] Authentication (publickey) failed. | |
| 11:43:37 | kevinbenton | good instance ID is 747e0481 | |
| 11:43:55 | johnthetubaguy | kevinbenton: maybe finding the server_id here will help, looks like the bad one was 31ca7a58-2a91-4893-bcb0-7c457023f6a1 | |
| 11:44:42 | kevinbenton | aye | |
| 11:44:47 | johnthetubaguy | kevinbenton: oh right, I see GET /2009-04-04/meta-data/public-keys/0/openssh-key | |
| 11:45:21 | kevinbenton | johnthetubaguy: so in the bad case whatever that first one is returning is telling it no keys | |
| 11:45:31 | johnthetubaguy | yeah, it seems it must be | |
| 11:46:16 | kevinbenton | is the preparation of the metadata asynchronous? | |
| 11:47:24 | johnthetubaguy | I think it should be all before the instance boots | |
| 11:47:28 | johnthetubaguy | but I am wondering that now too | |
| 11:51:22 | johnthetubaguy | kevinbenton: so the ssh key is in the compute logs | |
| 11:52:06 | johnthetubaguy | kevinbenton: 2017-02-28 11:47:44.387 5448 DEBUG nova.virt.libvirt.vif [req-b43e28a2-6040-4273-b21f-cfe8ac5f7fb6 tempest-TestGettingAddress-1009203826 tempest-TestGettingAddress-1009203826] vif_type=bridge instance= | |
| 11:52:32 | johnthetubaguy | includes uuid=31ca7a58-2a91-4893-bcb0-7c457023f6a1 | |
| 11:52:33 | johnthetubaguy | and | |
| 11:52:46 | johnthetubaguy | key_name='tempest-TestGettingAddress-1661300941' | |
| 11:52:55 | johnthetubaguy | it was a key_pair generated by nova, apparently | |
| 11:53:23 | johnthetubaguy | so I assume temptest is correctly remembering the new key | |
| 11:53:40 | johnthetubaguy | although clearly its not fetching it, so thats kinda irrelavant | |
| 11:54:51 | kevinbenton | johnthetubaguy: it seems we may be short on some relevant debug logs in the attempted key lookup in n-api | |
| 11:55:20 | johnthetubaguy | looks like the good one had key_name='tempest-TestGettingAddress-1661300941' | |
| 11:55:22 | johnthetubaguy | strange | |
| 11:55:37 | johnthetubaguy | kevinbenton: yeah, its probably worth adding that | |
| 11:55:52 | kevinbenton | johnthetubaguy: can you point me to that code and i'll push up a patch? | |
| 11:55:56 | sdague | kevinbenton: yeh, it's actually a seperate wsgi handler | |
| 11:56:20 | johnthetubaguy | kevinbenton: yeah, I think thats here: https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L273 | |
| 11:56:39 | johnthetubaguy | sdague: seems like its handled in process in this gate job though | |
| 11:56:52 | sdague | johnthetubaguy: ... no exactly | |
| 11:56:57 | sdague | not exactly | |
| 11:57:00 | kevinbenton | ha! https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L284 | |
| 11:57:16 | sdague | our n-api has the ability to run multiple worker pools | |
| 11:57:24 | johnthetubaguy | sdague: good point | |
| 11:57:27 | kevinbenton | sdague: will that present an issue with logging statements? | |
| 11:57:56 | sdague | kevinbenton: no, it's all going through a dedicated logger. It's kind of no different than other pools | |
| 11:58:03 | kevinbenton | akc | |
| 11:58:04 | kevinbenton | ack* | |
| 11:58:39 | kevinbenton | I think very few of our jobs use metadata so the poor linux bridge job is the canary in the coal mine | |
| 11:58:43 | sdague | honestly, it would probably be a sensible long term move to make this always stand on it's own and run on it's own | |
| 11:59:16 | sdague | but I take it that the growth of multiple apis servered from one system came from ec2 days | |
| 11:59:47 | asettle | johnthetubaguy: so, who should I be emailing for docs stuff for now? :) | |
| 12:00:18 | johnthetubaguy | asettle: unsure, mriedem_away is the best person to ask about that | |
| 12:00:29 | asettle | Yikes, haha, okay, we need to fix that | |
| 12:00:54 | johnthetubaguy | sdague: certainly worth doing when we shift to uwsgi or whatever it is | |
| 12:01:08 | sdague | there is a mode to run it dedicated today | |
| 12:01:17 | johnthetubaguy | ack | |
| 12:01:31 | sdague | I don't know if that would be less confusing if we defaulted that mode | |
| 12:01:58 | sdague | Also, while this code path never seems to be hit, the caching by remote IP address in here scares me | |
| 12:02:06 | sdague | given that is recycled | |
| 12:02:19 | sdague | https://github.com/openstack/nova/blob/37cd9a961b065a07352b49ee72394cb210d8838b/nova/api/metadata/handler.py#L56-L74 | |
| 12:03:09 | kevinbenton | sdague: ouch | |
| 12:03:24 | kevinbenton | sdague: oh, we shouldn't hit that though | |