| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-22 | |||
| 16:30:18 | stephenfin | thanks | |
| 17:22:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Remove nova-network remnants https://review.opendev.org/737329 | |
| 17:25:56 | openstackgerrit | Ghanshyam Mann proposed openstack/nova stable/stein: Make greande jobs n-v for EM and oldest stable https://review.opendev.org/737332 | |
| 17:27:48 | openstackgerrit | Ghanshyam Mann proposed openstack/nova stable/stein: Make greande jobs n-v for EM and oldest stable https://review.opendev.org/737332 | |
| 17:34:47 | sean-k-mooney | dansmith: you can increase the job timeout in the zull.yaml if you need to for the multistore job | |
| 17:35:05 | sean-k-mooney | dansmith: it looks like you glance api change made it this time https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/etc/glance/glance-api_conf.txt | |
| 17:35:27 | dansmith | yep, and it asked for the copy | |
| 17:35:32 | dansmith | I think something else likely broke, looking now | |
| 17:35:37 | dansmith | Jun 22 15:42:22.928857 ubuntu-bionic-rax-iad-0017311577 nova-compute[23701]: INFO nova.virt.libvirt.imagebackend [None req-ca48174a-0bf4-4341-8d45-fcf69cc9a3de tempest-DeleteServersAdminTestJSON-1752858908 tempest-DeleteServersAdminTestJSON-1752858908] Asking glance to copy image e6b1a7d0-ccd8-4be3-bef7-69c68fca4313 to our rbd store robust | |
| 17:36:14 | dansmith | Jun 22 15:52:23.076886 ubuntu-bionic-rax-iad-0017311577 nova-compute[23701]: ERROR nova.compute.manager [instance: 2cb1f8e2-a6a3-4f42-b6e2-de6823c71e25] nova.exception.ImageUnacceptable: Image e6b1a7d0-ccd8-4be3-bef7-69c68fca4313 is unacceptable: Copy to store robust timed out | |
| 17:36:53 | sean-k-mooney | it might be a slow node | |
| 17:37:12 | sean-k-mooney | you could relax some of the times outs for image/volume creation | |
| 17:37:23 | dansmith | it waited ten minutes | |
| 17:37:38 | dansmith | that should be more than long enough to copy a cirros image on any node I think | |
| 17:38:03 | sean-k-mooney | ya fair point :) | |
| 17:38:33 | sean-k-mooney | i was more thinking it was a slow host becaue it hit the 2 hour job time out | |
| 17:38:46 | sean-k-mooney | althougyh i guess enough 10 minute wait would have the same effect | |
| 17:39:28 | dansmith | I think it's just because each time we went to spawn an instance, it waited ten minutes before failing, | |
| 17:39:35 | dansmith | which linearized is enough to run the timeout | |
| 17:40:11 | dansmith | https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/screen-g-api.txt#7466 | |
| 17:40:18 | dansmith | glance was failing to update its own property I think | |
| 17:40:35 | sean-k-mooney | right this si the image convertion https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/screen-g-api.txt#440 | |
| 17:41:03 | sean-k-mooney | so it looks like the inital import conversion worked | |
| 17:42:04 | dansmith | the devstack conversion you mean? | |
| 17:42:14 | dansmith | had it not, nova wouldn't have even tried to boot on it, so yeah | |
| 17:42:27 | dansmith | and the devstack patch I have wouldn't have gotten past waiting for the image to go active | |
| 17:43:05 | sean-k-mooney | dansmith: yes the intial devstack conversion seam to have worked fine so the failure after after the qcow has been converted to raw and stored in teh file backedn | |
| 17:43:26 | dansmith | yep | |
| 17:43:54 | sean-k-mooney | well if nothing else i guess glance can now use your patch to test that... | |
| 17:44:43 | sean-k-mooney | os_glance_importing_to_stores seams like a strange name for a property on the image | |
| 17:45:08 | dansmith | that's the task status property | |
| 17:45:23 | sean-k-mooney | https://github.com/openstack/glance/blob/92492cf50461e214b777c707148886a8e87f340d/releasenotes/notes/import-multi-stores-3e781f2878b3134d.yaml#L25 yep | |
| 17:46:18 | sean-k-mooney | i guess the import-form-copy is modifying that to add the rbd store | |
| 17:48:27 | dansmith | right, the glance tasks modify that property to tell us what is happening | |
| 17:52:06 | sean-k-mooney | dansmith: i wonder if this could be related to who owns the image | |
| 17:52:20 | sean-k-mooney | devstack uploads it as admin correct | |
| 17:52:33 | sean-k-mooney | but tempest is running with its own tenats | |
| 17:52:42 | dansmith | well, that's the obvious thing, but the task should be using an admin context for this kind of metadata updating | |
| 17:52:47 | dansmith | and they say it should | |
| 17:52:49 | sean-k-mooney | so perhaps they do not have permission to modify that porperty | |
| 17:53:22 | sean-k-mooney | that would be the logical thing to do yes | |
| 17:53:29 | sean-k-mooney | but manybet its not | |
| 17:53:58 | sean-k-mooney | https://github.com/openstack/glance/commit/1754c9e2b085ba0fc37a4369488c92a40268997a add the copy image support so im just skiming it quickly to see what it does | |
| 18:01:23 | sean-k-mooney | home ok i dont see how the propery gets updated in that but i also dont know how glance works internally so its not suprising. | |
| 18:01:32 | sean-k-mooney | oh time for a call... | |
| 18:19:53 | dansmith | melwitt: a while back I asked about getting admin credentials for glance and you pointed me to something I ignored because I decided I didn't need admin | |
| 18:20:03 | dansmith | melwitt: do you remember that and if so can you point me again? | |
| 18:26:35 | melwitt | heh, sec | |
| 18:29:19 | melwitt | dansmith: it might have been this commit https://github.com/openstack/nova/commit/aab4b7a0e2504c04e08389145bcb1414dea63631 | |
| 18:29:58 | melwitt | just as an example of a place where we needed to use an admin cred to make a particular API call | |
| 18:34:08 | dansmith | melwitt: okay that's just a flag to the neutron client right? | |
| 18:34:47 | dansmith | I thought there was something more general | |
| 18:37:46 | melwitt | dansmith: yeah, I think when I linked you I was just saying, it is normal/expected for us to have to selectively use admin to call other APIs and that was a recent example of us doing it | |
| 18:38:33 | dansmith | oh, okay, that's common in a lot of places, yeah.. what I need is a way to get admin creds to talk to glance | |
| 18:38:54 | dansmith | I don't really know how we do that for neutron.. I think long ago we had credentials in our config, but that's gone now right? | |
| 18:41:28 | melwitt | I don't know off the top of my head. I thought we did have creds but I don't know about them being gone. I'm looking through the code now to see if it's obvious | |
| 18:42:34 | dansmith | I thought there was some service user thing we use now, but yeah I don't really know | |
| 18:53:03 | melwitt | based on this code block, there are supposed to be creds used from nova.conf https://github.com/openstack/nova/blob/f1ebc15dfc8ffb7f23b2cb9879f0ca9376931a90/nova/network/neutron.py#L191 | |
| 19:01:42 | melwitt | and here's a config file from a nova-next run showing what look to be service user creds for neutron and placement https://zuul.opendev.org/t/openstack/build/785733b6379b40a5982f710a62302c21/log/controller/logs/etc/nova/nova_cell1_conf.txt#40 | |
| 19:07:26 | dansmith | melwitt: sorry in three conversations here | |
| 19:07:38 | dansmith | melwitt: yeah, okay, I thought we had moved past that at some point, but it looks like not | |
| 19:07:40 | melwitt | np. I'm still gathering info | |
| 19:08:16 | melwitt | we implemented this https://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/use-service-tokens.html which says it should have docs for setting up the service user stuff in conf but I don't find any docs so far | |
| 19:08:23 | dansmith | glance is kinda half requiring admin to do the image copy-to-rbd thing.. if that's intentional, then we'll need admin creds for glance too, which really sucks | |
| 19:08:29 | dansmith | heh | |
| 19:08:56 | dansmith | maybe it's done and devstack is just still using the old method? | |
| 19:10:18 | melwitt | I'd think that unlikely... | |
| 19:10:31 | melwitt | this is what I find for a change that went into devstack to enable service tokens. https://review.opendev.org/#/c/409329/8/lib/nova | |
| 19:11:07 | melwitt | (I'm looking through https://review.opendev.org/#/q/topic:bp/use-service-tokens) | |
| 19:11:10 | dansmith | melwitt: come on, be optimistic with me! :P | |
| 19:11:16 | melwitt | lol :) | |
| 19:11:31 | melwitt | well, the service token stuff involves setting creds in conf | |
| 19:11:55 | melwitt | for the service user | |
| 19:11:58 | dansmith | hmm, okay maybe the service token still means we get creds, just not creds that are general purpose admins? | |
| 19:12:04 | dansmith | I thought it was better than that | |
| 19:12:08 | melwitt | how that's different I don't really know | |
| 19:13:14 | sean-k-mooney | dansmith: i dont think devstack conigure the service user stuff for us by default | |
| 19:13:33 | sean-k-mooney | i have not looked in a while but last time i did i did not see it | |
| 19:13:55 | melwitt | dansmith: so.... it "seems" like you would probably do a patch similar to this one https://review.opendev.org/410394 | |
| 19:14:41 | melwitt | that ^ adds the nova-neutron interaction. and the groundwork was added in a prior patch for the nova-cinder interaction https://review.opendev.org/397399 | |
| 19:15:09 | dansmith | ack yeah, okay | |
| 19:15:10 | dansmith | well, I guess I'll start by arguing that we shouldn't need admin to do this | |
| 19:15:49 | dansmith | melwitt: and just to stitch that stuff together, you think that when we do admin=True to the neutronclient, we're now getting the service user's auth token? | |
| 19:16:01 | dansmith | the logic on L135 there is a bit confusing | |
| 19:16:05 | melwitt | sean-k-mooney: we enable it in nova-next https://github.com/openstack/nova/blob/f1ebc15dfc8ffb7f23b2cb9879f0ca9376931a90/.zuul.yaml#L180 | |
| 19:16:23 | sean-k-mooney | melwitt: ah ok but not in dansmith's job https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/etc/nova/nova-cpu_conf.txt | |
| 19:16:54 | sean-k-mooney | there is no service_user group in the config | |
| 19:17:07 | melwitt | yeah you'd have to set the env var for the job | |
| 19:17:58 | sean-k-mooney | could you just add the glance credetials like we do for neutron https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/etc/nova/nova-cpu_conf.txt#43-50 | |
| 19:18:36 | dansmith | sean-k-mooney: that's what melwitt said above | |
| 19:18:40 | sean-k-mooney | service_user support was more fo the case where i am doing a long running request but my user token expired so we fallback to a admin service user instead | |
| 19:18:59 | dansmith | and I'm guessing the answer is yes, but I'd like to not have to do that (meaning not need to have admin for this) | |
| 19:19:57 | sean-k-mooney | ya given you dont own the image however im not surprised that glance is unhappy | |
| 19:20:14 | sean-k-mooney | you could argue that if its a public image then maybe this should be allowed | |
| 19:20:17 | melwitt | dansmith: yeah... agreed it looks confusing. but I think yeah, passing admin=True is having it load the auth plugin from conf, which presumably will pick up the [service_user] config section | |
| 19:20:34 | dansmith | sean-k-mooney: it's public and the API is letting me do the operation | |
| 19:20:42 | sean-k-mooney | melwitt: i dont think it will by default | |
| 19:21:08 | sean-k-mooney | dansmith: as in the api is accepting the import | |
| 19:21:14 | dansmith | right | |
| 19:21:22 | sean-k-mooney | ya so it feels like a glance but | |