| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 19:26:17 | mriedem | it's a POST, | |
| 19:26:24 | mriedem | but i'm not sure where the metadata service returns | |
| 19:26:34 | openstackgerrit | Merged openstack/nova master: api-ref: fix type for hypervisor_marker https://review.openstack.org/471970 | |
| 19:26:41 | kevinbenton | ihrachys: hey | |
| 19:28:07 | kevinbenton | ihrachys: looking at backlog. l2-agent never deletes a tap device, that's left up to nova | |
| 19:28:12 | kevinbenton | mriedem: ^^ | |
| 19:28:23 | kevinbenton | but it looks like you already found that | |
| 19:28:26 | kevinbenton | anything i can help with? | |
| 19:32:59 | mriedem | dansmith: i guess the metadata POST is synchronous | |
| 19:36:47 | mriedem | dansmith: aha http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/screen-n-api.txt.gz#_Jun_08_18_36_06_549748 | |
| 19:36:53 | mriedem | the cache fs us again | |
| 19:37:25 | mriedem | we update inst.system_metadata, then tempest calls the rest api to get the password, and it looks like things are going through the metadata cache | |
| 19:37:35 | mriedem | not sure why we wouldn't update the cache when we update/set the password... | |
| 19:38:10 | dansmith | but that's still unrelated right? | |
| 19:38:11 | dansmith | but yeah, that metadata stuff needs fixin' | |
| 19:38:21 | mriedem | wtf, we're getting an InstanceNotFound | |
| 19:38:37 | mriedem | oooooo | |
| 19:38:47 | mriedem | it is cells | |
| 19:38:49 | mriedem | we're not targeted | |
| 19:38:50 | mriedem | mofo | |
| 19:38:57 | mriedem | opening a bug | |
| 19:39:36 | dansmith | we shouldn't be targeted | |
| 19:39:47 | mriedem | actually the instance should be targeted when we look it up | |
| 19:40:03 | mriedem | i mean, we look up the mapping and then target the instance don't we? | |
| 19:40:14 | dansmith | metadata should be a cell thing so it shouldn't be using target_cell, is my point | |
| 19:40:28 | dansmith | maybe that's the problem | |
| 19:40:44 | dansmith | does metadata run as a separate process in devstack? | |
| 19:41:01 | mriedem | not here, it's part of nova-api | |
| 19:41:25 | openstackgerrit | Merged openstack/nova master: Add docstring for test_limit_check_project_and_user_zero_values https://review.openstack.org/472287 | |
| 19:42:50 | mriedem | https://bugs.launchpad.net/nova/+bug/1696843 | |
| 19:42:50 | openstack | Launchpad bug 1696843 in OpenStack Compute (nova) "metadata: POST /openstack/2013-10-17/password fails with InstanceNotFound" [Undecided,New] | |
| 19:43:44 | mriedem | kevinbenton: i've got a debug patch up to see if i can recreate in a neutron patch and get some dumps of the related domain xml | |
| 19:43:55 | dansmith | mriedem: well, that's going to be a problem then | |
| 19:44:02 | mriedem | and ihar has a patch up with a revert of the lb tap change to see if that is any different | |
| 19:45:01 | mriedem | dansmith: right here https://github.com/openstack/nova/blob/b535c7432d02366f11fad7d1b1a3e3a8b8ac5415/nova/api/metadata/password.py#L70 | |
| 19:45:09 | mriedem | why wouldn't we lookup the instance mapping via that uuid | |
| 19:45:16 | mriedem | and then target the get_by_uuid to the cell | |
| 19:45:19 | dansmith | mriedem: because metadata is in the cell and we can't do that | |
| 19:45:29 | mriedem | i thought it was global | |
| 19:45:43 | mriedem | actually i don't think i've ever heard of anyone ever talk about the metadata service wrt cells v2 | |
| 19:46:10 | dansmith | not sure why it would be global, | |
| 19:46:11 | dansmith | but also, | |
| 19:46:19 | dansmith | not sure from a neutron perspective | |
| 19:46:28 | dansmith | I always think of the world like nova net multihost | |
| 19:46:40 | dansmith | maybe we can ask kevinbenton since he's here | |
| 19:46:43 | mriedem | why wouldn't it be global if the api is global? | |
| 19:47:00 | mriedem | i guess because you want the guests confined to the cell | |
| 19:47:02 | mriedem | ? | |
| 19:47:09 | dansmith | hmm, we already target in metadata one place | |
| 19:47:14 | kevinbenton | ask me what? | |
| 19:47:15 | mriedem | yeah... | |
| 19:47:40 | mriedem | if metadata can't be global then it's going to have to know what cell it's in | |
| 19:47:41 | dansmith | mriedem: in multihost you run one per compute node so you scale 1:1 with computes | |
| 19:48:10 | dansmith | mriedem: I don't think I understand why that is | |
| 19:48:25 | dansmith | mriedem: however, question would be how the neutron proxy thing finds the metadata api | |
| 19:48:29 | dansmith | kevinbenton: ^ | |
| 19:48:38 | mriedem | it's configured in neutron.conf i though | |
| 19:48:40 | mriedem | *thought | |
| 19:48:41 | kevinbenton | dansmith: ah, it's just a hardcoded config | |
| 19:48:47 | kevinbenton | dansmith: of a single http endpoint | |
| 19:48:49 | dansmith | but neutron config per compute node right? | |
| 19:48:53 | mriedem | no | |
| 19:49:05 | mriedem | well i can't answer that | |
| 19:49:05 | kevinbenton | it's on the network node that runs the proxy | |
| 19:49:14 | kevinbenton | so not compute node | |
| 19:49:20 | dansmith | kevinbenton: so is it a single config for the whole deployment? | |
| 19:49:30 | kevinbenton | dansmith: essentially yes | |
| 19:49:37 | kevinbenton | dansmith: you could have different configs per network node | |
| 19:49:40 | dansmith | okay, not sure that's a good thing, although it does make this easier | |
| 19:49:44 | kevinbenton | but i don't think that would do anything useful | |
| 19:49:52 | mriedem | kevinbenton: where is that in http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/etc/neutron/neutron.conf.txt.gz ? | |
| 19:49:53 | kevinbenton | since networks nodes aren't correlated with cells | |
| 19:49:57 | mriedem | oh nvm http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/etc/neutron/metadata_agent.ini.txt.gz | |
| 19:50:01 | mriedem | nova_metadata_ip = 10.40.184.140 | |
| 19:50:05 | kevinbenton | mriedem: +1 | |
| 19:50:14 | dansmith | from a scale point of view I mean, but since we already effectively have it global I guess that's a totally different conversation | |
| 19:50:43 | mriedem | don't people already disable metadata service b/c of perf issues, and security | |
| 19:51:00 | mriedem | i know that's irrelevant | |
| 19:51:01 | dansmith | depends on their goals, but some do yeah | |
| 19:51:34 | dansmith | alright anyway, I'll fix that up | |
| 19:51:55 | openstackgerrit | Dan Smith proposed openstack/nova master: Remove old service version check for mitaka https://review.openstack.org/472395 | |
| 19:52:28 | mriedem | i suppose this meta bug has to be backported | |
| 19:52:58 | mriedem | although i'm only seeing it in the multi-cell devstack change | |
| 19:56:51 | kevinbenton | dansmith, mriedem: is there a way we can know which cell to send something to? | |
| 19:57:09 | kevinbenton | we could alter the proxy logic to route in some manner because it does do a port lookup to inject headers | |
| 19:57:18 | dansmith | mriedem: I don't think we need to backport it | |
| 19:57:25 | dansmith | kevinbenton: you don't and shouldn't know any of that | |
| 19:58:33 | kevinbenton | dansmith: ok, just checking if this is an issue we need to fix on our metadata proxy logic | |
| 19:59:16 | mriedem | i wonder how cells v1 people are doing the metadata service | |
| 19:59:17 | dansmith | kevinbenton: no it sounds like you're effectively forcing a global metadata server now, which makes a fix for this easy on our end | |
| 19:59:30 | dansmith | mriedem: not sure it works in cellsv1, or they just run it in a cell I'd imagine | |
| 19:59:44 | mriedem | i'll ask in ops | |
| 19:59:51 | kevinbenton | just put a load balancer in front of it that fans every request out to a cell until it doesn't get a 404 :) | |
| 20:02:28 | mriedem | you don't get a 404 here, | |
| 20:02:30 | mriedem | you get a 500 :( | |
| 20:02:41 | mriedem | we probably also need to fix that | |
| 20:02:56 | mriedem | http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/screen-n-api.txt.gz#_Jun_08_18_36_06_549748 | |
| 20:03:12 | mriedem | 10.1.0.11,10.40.184.140 "POST /openstack/2013-10-17/password HTTP/1.1" status: 500 len: 139 time: 0.0445061: InstanceNotFound: Instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f could not be found. | |
| 20:03:53 | mriedem | i'll open a separate bug for that | |
| 20:04:03 | mriedem | which is backportable | |