| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-08 | |||
| 19:38:21 | mriedem | wtf, we're getting an InstanceNotFound | |
| 19:38:37 | mriedem | oooooo | |
| 19:38:47 | mriedem | it is cells | |
| 19:38:49 | mriedem | we're not targeted | |
| 19:38:50 | mriedem | mofo | |
| 19:38:57 | mriedem | opening a bug | |
| 19:39:36 | dansmith | we shouldn't be targeted | |
| 19:39:47 | mriedem | actually the instance should be targeted when we look it up | |
| 19:40:03 | mriedem | i mean, we look up the mapping and then target the instance don't we? | |
| 19:40:14 | dansmith | metadata should be a cell thing so it shouldn't be using target_cell, is my point | |
| 19:40:28 | dansmith | maybe that's the problem | |
| 19:40:44 | dansmith | does metadata run as a separate process in devstack? | |
| 19:41:01 | mriedem | not here, it's part of nova-api | |
| 19:41:25 | openstackgerrit | Merged openstack/nova master: Add docstring for test_limit_check_project_and_user_zero_values https://review.openstack.org/472287 | |
| 19:42:50 | openstack | Launchpad bug 1696843 in OpenStack Compute (nova) "metadata: POST /openstack/2013-10-17/password fails with InstanceNotFound" [Undecided,New] | |
| 19:42:50 | mriedem | https://bugs.launchpad.net/nova/+bug/1696843 | |
| 19:43:44 | mriedem | kevinbenton: i've got a debug patch up to see if i can recreate in a neutron patch and get some dumps of the related domain xml | |
| 19:43:55 | dansmith | mriedem: well, that's going to be a problem then | |
| 19:44:02 | mriedem | and ihar has a patch up with a revert of the lb tap change to see if that is any different | |
| 19:45:01 | mriedem | dansmith: right here https://github.com/openstack/nova/blob/b535c7432d02366f11fad7d1b1a3e3a8b8ac5415/nova/api/metadata/password.py#L70 | |
| 19:45:09 | mriedem | why wouldn't we lookup the instance mapping via that uuid | |
| 19:45:16 | mriedem | and then target the get_by_uuid to the cell | |
| 19:45:19 | dansmith | mriedem: because metadata is in the cell and we can't do that | |
| 19:45:29 | mriedem | i thought it was global | |
| 19:45:43 | mriedem | actually i don't think i've ever heard of anyone ever talk about the metadata service wrt cells v2 | |
| 19:46:10 | dansmith | not sure why it would be global, | |
| 19:46:11 | dansmith | but also, | |
| 19:46:19 | dansmith | not sure from a neutron perspective | |
| 19:46:28 | dansmith | I always think of the world like nova net multihost | |
| 19:46:40 | dansmith | maybe we can ask kevinbenton since he's here | |
| 19:46:43 | mriedem | why wouldn't it be global if the api is global? | |
| 19:47:00 | mriedem | i guess because you want the guests confined to the cell | |
| 19:47:02 | mriedem | ? | |
| 19:47:09 | dansmith | hmm, we already target in metadata one place | |
| 19:47:14 | kevinbenton | ask me what? | |
| 19:47:15 | mriedem | yeah... | |
| 19:47:40 | mriedem | if metadata can't be global then it's going to have to know what cell it's in | |
| 19:47:41 | dansmith | mriedem: in multihost you run one per compute node so you scale 1:1 with computes | |
| 19:48:10 | dansmith | mriedem: I don't think I understand why that is | |
| 19:48:25 | dansmith | mriedem: however, question would be how the neutron proxy thing finds the metadata api | |
| 19:48:29 | dansmith | kevinbenton: ^ | |
| 19:48:38 | mriedem | it's configured in neutron.conf i though | |
| 19:48:40 | mriedem | *thought | |
| 19:48:41 | kevinbenton | dansmith: ah, it's just a hardcoded config | |
| 19:48:47 | kevinbenton | dansmith: of a single http endpoint | |
| 19:48:49 | dansmith | but neutron config per compute node right? | |
| 19:48:53 | mriedem | no | |
| 19:49:05 | kevinbenton | it's on the network node that runs the proxy | |
| 19:49:05 | mriedem | well i can't answer that | |
| 19:49:14 | kevinbenton | so not compute node | |
| 19:49:20 | dansmith | kevinbenton: so is it a single config for the whole deployment? | |
| 19:49:30 | kevinbenton | dansmith: essentially yes | |
| 19:49:37 | kevinbenton | dansmith: you could have different configs per network node | |
| 19:49:40 | dansmith | okay, not sure that's a good thing, although it does make this easier | |
| 19:49:44 | kevinbenton | but i don't think that would do anything useful | |
| 19:49:52 | mriedem | kevinbenton: where is that in http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/etc/neutron/neutron.conf.txt.gz ? | |
| 19:49:53 | kevinbenton | since networks nodes aren't correlated with cells | |
| 19:49:57 | mriedem | oh nvm http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/etc/neutron/metadata_agent.ini.txt.gz | |
| 19:50:01 | mriedem | nova_metadata_ip = 10.40.184.140 | |
| 19:50:05 | kevinbenton | mriedem: +1 | |
| 19:50:14 | dansmith | from a scale point of view I mean, but since we already effectively have it global I guess that's a totally different conversation | |
| 19:50:43 | mriedem | don't people already disable metadata service b/c of perf issues, and security | |
| 19:51:00 | mriedem | i know that's irrelevant | |
| 19:51:01 | dansmith | depends on their goals, but some do yeah | |
| 19:51:34 | dansmith | alright anyway, I'll fix that up | |
| 19:51:55 | openstackgerrit | Dan Smith proposed openstack/nova master: Remove old service version check for mitaka https://review.openstack.org/472395 | |
| 19:52:28 | mriedem | i suppose this meta bug has to be backported | |
| 19:52:58 | mriedem | although i'm only seeing it in the multi-cell devstack change | |
| 19:56:51 | kevinbenton | dansmith, mriedem: is there a way we can know which cell to send something to? | |
| 19:57:09 | kevinbenton | we could alter the proxy logic to route in some manner because it does do a port lookup to inject headers | |
| 19:57:18 | dansmith | mriedem: I don't think we need to backport it | |
| 19:57:25 | dansmith | kevinbenton: you don't and shouldn't know any of that | |
| 19:58:33 | kevinbenton | dansmith: ok, just checking if this is an issue we need to fix on our metadata proxy logic | |
| 19:59:16 | mriedem | i wonder how cells v1 people are doing the metadata service | |
| 19:59:17 | dansmith | kevinbenton: no it sounds like you're effectively forcing a global metadata server now, which makes a fix for this easy on our end | |
| 19:59:30 | dansmith | mriedem: not sure it works in cellsv1, or they just run it in a cell I'd imagine | |
| 19:59:44 | mriedem | i'll ask in ops | |
| 19:59:51 | kevinbenton | just put a load balancer in front of it that fans every request out to a cell until it doesn't get a 404 :) | |
| 20:02:28 | mriedem | you don't get a 404 here, | |
| 20:02:30 | mriedem | you get a 500 :( | |
| 20:02:41 | mriedem | we probably also need to fix that | |
| 20:02:56 | mriedem | http://logs.openstack.org/94/436094/33/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/bed0e6a/logs/screen-n-api.txt.gz#_Jun_08_18_36_06_549748 | |
| 20:03:12 | mriedem | 10.1.0.11,10.40.184.140 "POST /openstack/2013-10-17/password HTTP/1.1" status: 500 len: 139 time: 0.0445061: InstanceNotFound: Instance 6cd26f78-9c8a-4b37-8c62-38ad003a095f could not be found. | |
| 20:03:53 | mriedem | i'll open a separate bug for that | |
| 20:04:03 | mriedem | which is backportable | |
| 20:05:58 | openstack | Launchpad bug 1696848 in OpenStack Compute (nova) "POST /openstack/2013-10-17/password returns 500 if instance is not found" [Medium,Triaged] | |
| 20:05:58 | mriedem | https://bugs.launchpad.net/nova/+bug/1696848 | |
| 20:08:11 | mriedem | claudiub: are you doing anything with this? https://blueprints.launchpad.net/nova/+spec/hyperv-sriov | |
| 20:08:16 | mriedem | else i'm going to defer it to queens | |
| 20:13:12 | mriedem | gdi, you know what's awesome, | |
| 20:13:31 | mriedem | overriding a method and tests start failing, you spend too much time trying to figure out why, | |
| 20:13:43 | mriedem | and realize you're not returning the results of the method you're now delegating too | |
| 20:13:45 | mriedem | *to | |
| 20:18:10 | openstackgerrit | Maciej Kucia proposed openstack/nova master: libvirt: Extract method _guest_add_spice_channel https://review.openstack.org/470787 | |
| 20:18:11 | openstackgerrit | Maciej Kucia proposed openstack/nova master: libvirt: Extract method _guest_add_memory_balloon https://review.openstack.org/470786 | |
| 20:18:11 | openstackgerrit | Maciej Kucia proposed openstack/nova master: libvirt: Extract method _guest_add_video_device https://review.openstack.org/470783 | |
| 20:18:12 | openstackgerrit | Maciej Kucia proposed openstack/nova master: libvirt: Extract method _guest_add_pci_devices https://review.openstack.org/470784 | |
| 20:18:12 | openstackgerrit | Maciej Kucia proposed openstack/nova master: libvirt: Extract method _guest_add_watchdog_action https://review.openstack.org/470785 | |
| 20:18:56 | melwitt | does anyone know if it matter whether you give a sqlalchemy in_() call a list or a set? is there any benefit in a set vs a list? | |
| 20:19:01 | melwitt | *matters | |