| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-22 | |||
| 13:42:54 | dansmith | right, so I don't see that we're making any different assumptions | |
| 13:43:04 | tbarron | sean-k-mooney: so the cinder lvm backend is useful for testing iscsi but not so much for production deployments | |
| 13:43:21 | dansmith | gibi: well, I disagree because I think that a property should explode for a known condition | |
| 13:43:37 | dansmith | gibi: but make it not a property (and rename it) and you can have that structure | |
| 13:43:48 | gibi | dansmith: I accep that I'm ready to make that an internal helper instead of a public property | |
| 13:44:00 | dansmith | I think a property shouild /not/ explode I meant | |
| 13:44:07 | gibi | yeah, I agree ^^ | |
| 13:45:09 | gibi | just to make sure I understand your point. Is it OK for you if change the property to an private helper method? | |
| 13:45:46 | dansmith | I don't like it, but it addresses the problem I had with PS2 | |
| 13:46:40 | gibi | why don't you like it? | |
| 13:49:00 | dansmith | well, because as it is, the property has utility beyond what you're doing here. You're just changing it to "should I report zero for cache" which is a single conditional and might as well just be in the if statement of the get_disk_usage() | |
| 13:49:14 | dansmith | doesn't seem worth it being a helper to me | |
| 13:49:52 | dansmith | but all I really meant is that _I_ would keep it the way it is in PS5, but it matters to me less than you, so you should change it | |
| 13:50:15 | dansmith | what matters to me is not having that should-be-useful-but-dangerous public property | |
| 13:52:42 | gibi | dansmith: thanks | |
| 14:04:43 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184 | |
| 14:25:46 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184 | |
| 14:27:16 | openstackgerrit | Elod Illes proposed openstack/nova stable/train: Check cherry-pick hashes in pep8 tox target https://review.opendev.org/737279 | |
| 15:08:06 | jsuchome | hey dansmith ... regular reminder about https://review.opendev.org/#/c/574301 once you have time... | |
| 15:28:00 | dansmith | jsuchome: I know, I haven't forgotten | |
| 15:45:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Guard against missing image cache directory https://review.opendev.org/736964 | |
| 16:29:46 | stephenfin | melwitt: could you look at https://review.opendev.org/708617 too? | |
| 16:30:12 | melwitt | stephenfin: sure, will do | |
| 16:30:18 | stephenfin | thanks | |
| 17:22:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Remove nova-network remnants https://review.opendev.org/737329 | |
| 17:25:56 | openstackgerrit | Ghanshyam Mann proposed openstack/nova stable/stein: Make greande jobs n-v for EM and oldest stable https://review.opendev.org/737332 | |
| 17:27:48 | openstackgerrit | Ghanshyam Mann proposed openstack/nova stable/stein: Make greande jobs n-v for EM and oldest stable https://review.opendev.org/737332 | |
| 17:34:47 | sean-k-mooney | dansmith: you can increase the job timeout in the zull.yaml if you need to for the multistore job | |
| 17:35:05 | sean-k-mooney | dansmith: it looks like you glance api change made it this time https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/etc/glance/glance-api_conf.txt | |
| 17:35:27 | dansmith | yep, and it asked for the copy | |
| 17:35:32 | dansmith | I think something else likely broke, looking now | |
| 17:35:37 | dansmith | Jun 22 15:42:22.928857 ubuntu-bionic-rax-iad-0017311577 nova-compute[23701]: INFO nova.virt.libvirt.imagebackend [None req-ca48174a-0bf4-4341-8d45-fcf69cc9a3de tempest-DeleteServersAdminTestJSON-1752858908 tempest-DeleteServersAdminTestJSON-1752858908] Asking glance to copy image e6b1a7d0-ccd8-4be3-bef7-69c68fca4313 to our rbd store robust | |
| 17:36:14 | dansmith | Jun 22 15:52:23.076886 ubuntu-bionic-rax-iad-0017311577 nova-compute[23701]: ERROR nova.compute.manager [instance: 2cb1f8e2-a6a3-4f42-b6e2-de6823c71e25] nova.exception.ImageUnacceptable: Image e6b1a7d0-ccd8-4be3-bef7-69c68fca4313 is unacceptable: Copy to store robust timed out | |
| 17:36:53 | sean-k-mooney | it might be a slow node | |
| 17:37:12 | sean-k-mooney | you could relax some of the times outs for image/volume creation | |
| 17:37:23 | dansmith | it waited ten minutes | |
| 17:37:38 | dansmith | that should be more than long enough to copy a cirros image on any node I think | |
| 17:38:03 | sean-k-mooney | ya fair point :) | |
| 17:38:33 | sean-k-mooney | i was more thinking it was a slow host becaue it hit the 2 hour job time out | |
| 17:38:46 | sean-k-mooney | althougyh i guess enough 10 minute wait would have the same effect | |
| 17:39:28 | dansmith | I think it's just because each time we went to spawn an instance, it waited ten minutes before failing, | |
| 17:39:35 | dansmith | which linearized is enough to run the timeout | |
| 17:40:11 | dansmith | https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/screen-g-api.txt#7466 | |
| 17:40:18 | dansmith | glance was failing to update its own property I think | |
| 17:40:35 | sean-k-mooney | right this si the image convertion https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/screen-g-api.txt#440 | |
| 17:41:03 | sean-k-mooney | so it looks like the inital import conversion worked | |
| 17:42:04 | dansmith | the devstack conversion you mean? | |
| 17:42:14 | dansmith | had it not, nova wouldn't have even tried to boot on it, so yeah | |
| 17:42:27 | dansmith | and the devstack patch I have wouldn't have gotten past waiting for the image to go active | |
| 17:43:05 | sean-k-mooney | dansmith: yes the intial devstack conversion seam to have worked fine so the failure after after the qcow has been converted to raw and stored in teh file backedn | |
| 17:43:26 | dansmith | yep | |
| 17:43:54 | sean-k-mooney | well if nothing else i guess glance can now use your patch to test that... | |
| 17:44:43 | sean-k-mooney | os_glance_importing_to_stores seams like a strange name for a property on the image | |
| 17:45:08 | dansmith | that's the task status property | |
| 17:45:23 | sean-k-mooney | https://github.com/openstack/glance/blob/92492cf50461e214b777c707148886a8e87f340d/releasenotes/notes/import-multi-stores-3e781f2878b3134d.yaml#L25 yep | |
| 17:46:18 | sean-k-mooney | i guess the import-form-copy is modifying that to add the rbd store | |
| 17:48:27 | dansmith | right, the glance tasks modify that property to tell us what is happening | |
| 17:52:06 | sean-k-mooney | dansmith: i wonder if this could be related to who owns the image | |
| 17:52:20 | sean-k-mooney | devstack uploads it as admin correct | |
| 17:52:33 | sean-k-mooney | but tempest is running with its own tenats | |
| 17:52:42 | dansmith | well, that's the obvious thing, but the task should be using an admin context for this kind of metadata updating | |
| 17:52:47 | dansmith | and they say it should | |
| 17:52:49 | sean-k-mooney | so perhaps they do not have permission to modify that porperty | |
| 17:53:22 | sean-k-mooney | that would be the logical thing to do yes | |
| 17:53:29 | sean-k-mooney | but manybet its not | |
| 17:53:58 | sean-k-mooney | https://github.com/openstack/glance/commit/1754c9e2b085ba0fc37a4369488c92a40268997a add the copy image support so im just skiming it quickly to see what it does | |
| 18:01:23 | sean-k-mooney | home ok i dont see how the propery gets updated in that but i also dont know how glance works internally so its not suprising. | |
| 18:01:32 | sean-k-mooney | oh time for a call... | |
| 18:19:53 | dansmith | melwitt: a while back I asked about getting admin credentials for glance and you pointed me to something I ignored because I decided I didn't need admin | |
| 18:20:03 | dansmith | melwitt: do you remember that and if so can you point me again? | |
| 18:26:35 | melwitt | heh, sec | |
| 18:29:19 | melwitt | dansmith: it might have been this commit https://github.com/openstack/nova/commit/aab4b7a0e2504c04e08389145bcb1414dea63631 | |
| 18:29:58 | melwitt | just as an example of a place where we needed to use an admin cred to make a particular API call | |
| 18:34:08 | dansmith | melwitt: okay that's just a flag to the neutron client right? | |
| 18:34:47 | dansmith | I thought there was something more general | |
| 18:37:46 | melwitt | dansmith: yeah, I think when I linked you I was just saying, it is normal/expected for us to have to selectively use admin to call other APIs and that was a recent example of us doing it | |
| 18:38:33 | dansmith | oh, okay, that's common in a lot of places, yeah.. what I need is a way to get admin creds to talk to glance | |
| 18:38:54 | dansmith | I don't really know how we do that for neutron.. I think long ago we had credentials in our config, but that's gone now right? | |
| 18:41:28 | melwitt | I don't know off the top of my head. I thought we did have creds but I don't know about them being gone. I'm looking through the code now to see if it's obvious | |
| 18:42:34 | dansmith | I thought there was some service user thing we use now, but yeah I don't really know | |
| 18:53:03 | melwitt | based on this code block, there are supposed to be creds used from nova.conf https://github.com/openstack/nova/blob/f1ebc15dfc8ffb7f23b2cb9879f0ca9376931a90/nova/network/neutron.py#L191 | |
| 19:01:42 | melwitt | and here's a config file from a nova-next run showing what look to be service user creds for neutron and placement https://zuul.opendev.org/t/openstack/build/785733b6379b40a5982f710a62302c21/log/controller/logs/etc/nova/nova_cell1_conf.txt#40 | |
| 19:07:26 | dansmith | melwitt: sorry in three conversations here | |
| 19:07:38 | dansmith | melwitt: yeah, okay, I thought we had moved past that at some point, but it looks like not | |
| 19:07:40 | melwitt | np. I'm still gathering info | |
| 19:08:16 | melwitt | we implemented this https://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/use-service-tokens.html which says it should have docs for setting up the service user stuff in conf but I don't find any docs so far | |
| 19:08:23 | dansmith | glance is kinda half requiring admin to do the image copy-to-rbd thing.. if that's intentional, then we'll need admin creds for glance too, which really sucks | |
| 19:08:29 | dansmith | heh | |
| 19:08:56 | dansmith | maybe it's done and devstack is just still using the old method? | |
| 19:10:18 | melwitt | I'd think that unlikely... | |
| 19:10:31 | melwitt | this is what I find for a change that went into devstack to enable service tokens. https://review.opendev.org/#/c/409329/8/lib/nova | |
| 19:11:07 | melwitt | (I'm looking through https://review.opendev.org/#/q/topic:bp/use-service-tokens) | |
| 19:11:10 | dansmith | melwitt: come on, be optimistic with me! :P | |
| 19:11:16 | melwitt | lol :) | |
| 19:11:31 | melwitt | well, the service token stuff involves setting creds in conf | |
| 19:11:55 | melwitt | for the service user | |
| 19:11:58 | dansmith | hmm, okay maybe the service token still means we get creds, just not creds that are general purpose admins? | |
| 19:12:04 | dansmith | I thought it was better than that | |
| 19:12:08 | melwitt | how that's different I don't really know | |
| 19:13:14 | sean-k-mooney | dansmith: i dont think devstack conigure the service user stuff for us by default | |