Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-06-22
13:43:48 gibi dansmith: I accep that I'm ready to make that an internal helper instead of a public property
13:44:00 dansmith I think a property shouild /not/ explode I meant
13:44:07 gibi yeah, I agree ^^
13:45:09 gibi just to make sure I understand your point. Is it OK for you if change the property to an private helper method?
13:45:46 dansmith I don't like it, but it addresses the problem I had with PS2
13:46:40 gibi why don't you like it?
13:49:00 dansmith well, because as it is, the property has utility beyond what you're doing here. You're just changing it to "should I report zero for cache" which is a single conditional and might as well just be in the if statement of the get_disk_usage()
13:49:14 dansmith doesn't seem worth it being a helper to me
13:49:52 dansmith but all I really meant is that _I_ would keep it the way it is in PS5, but it matters to me less than you, so you should change it
13:50:15 dansmith what matters to me is not having that should-be-useful-but-dangerous public property
13:52:42 gibi dansmith: thanks
14:04:43 openstackgerrit Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184
14:25:46 openstackgerrit Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184
14:27:16 openstackgerrit Elod Illes proposed openstack/nova stable/train: Check cherry-pick hashes in pep8 tox target https://review.opendev.org/737279
15:08:06 jsuchome hey dansmith ... regular reminder about https://review.opendev.org/#/c/574301 once you have time...
15:28:00 dansmith jsuchome: I know, I haven't forgotten
15:45:46 openstackgerrit Balazs Gibizer proposed openstack/nova master: Guard against missing image cache directory https://review.opendev.org/736964
16:29:46 stephenfin melwitt: could you look at https://review.opendev.org/708617 too?
16:30:12 melwitt stephenfin: sure, will do
16:30:18 stephenfin thanks
17:22:33 openstackgerrit Stephen Finucane proposed openstack/nova master: fakelibvirt: Remove nova-network remnants https://review.opendev.org/737329
17:25:56 openstackgerrit Ghanshyam Mann proposed openstack/nova stable/stein: Make greande jobs n-v for EM and oldest stable https://review.opendev.org/737332
17:27:48 openstackgerrit Ghanshyam Mann proposed openstack/nova stable/stein: Make greande jobs n-v for EM and oldest stable https://review.opendev.org/737332
17:34:47 sean-k-mooney dansmith: you can increase the job timeout in the zull.yaml if you need to for the multistore job
17:35:05 sean-k-mooney dansmith: it looks like you glance api change made it this time https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/etc/glance/glance-api_conf.txt
17:35:27 dansmith yep, and it asked for the copy
17:35:32 dansmith I think something else likely broke, looking now
17:35:37 dansmith Jun 22 15:42:22.928857 ubuntu-bionic-rax-iad-0017311577 nova-compute[23701]: INFO nova.virt.libvirt.imagebackend [None req-ca48174a-0bf4-4341-8d45-fcf69cc9a3de tempest-DeleteServersAdminTestJSON-1752858908 tempest-DeleteServersAdminTestJSON-1752858908] Asking glance to copy image e6b1a7d0-ccd8-4be3-bef7-69c68fca4313 to our rbd store robust
17:36:14 dansmith Jun 22 15:52:23.076886 ubuntu-bionic-rax-iad-0017311577 nova-compute[23701]: ERROR nova.compute.manager [instance: 2cb1f8e2-a6a3-4f42-b6e2-de6823c71e25] nova.exception.ImageUnacceptable: Image e6b1a7d0-ccd8-4be3-bef7-69c68fca4313 is unacceptable: Copy to store robust timed out
17:36:53 sean-k-mooney it might be a slow node
17:37:12 sean-k-mooney you could relax some of the times outs for image/volume creation
17:37:23 dansmith it waited ten minutes
17:37:38 dansmith that should be more than long enough to copy a cirros image on any node I think
17:38:03 sean-k-mooney ya fair point :)
17:38:33 sean-k-mooney i was more thinking it was a slow host becaue it hit the 2 hour job time out
17:38:46 sean-k-mooney althougyh i guess enough 10 minute wait would have the same effect
17:39:28 dansmith I think it's just because each time we went to spawn an instance, it waited ten minutes before failing,
17:39:35 dansmith which linearized is enough to run the timeout
17:40:11 dansmith https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/screen-g-api.txt#7466
17:40:18 dansmith glance was failing to update its own property I think
17:40:35 sean-k-mooney right this si the image convertion https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/screen-g-api.txt#440
17:41:03 sean-k-mooney so it looks like the inital import conversion worked
17:42:04 dansmith the devstack conversion you mean?
17:42:14 dansmith had it not, nova wouldn't have even tried to boot on it, so yeah
17:42:27 dansmith and the devstack patch I have wouldn't have gotten past waiting for the image to go active
17:43:05 sean-k-mooney dansmith: yes the intial devstack conversion seam to have worked fine so the failure after after the qcow has been converted to raw and stored in teh file backedn
17:43:26 dansmith yep
17:43:54 sean-k-mooney well if nothing else i guess glance can now use your patch to test that...
17:44:43 sean-k-mooney os_glance_importing_to_stores seams like a strange name for a property on the image
17:45:08 dansmith that's the task status property
17:45:23 sean-k-mooney https://github.com/openstack/glance/blob/92492cf50461e214b777c707148886a8e87f340d/releasenotes/notes/import-multi-stores-3e781f2878b3134d.yaml#L25 yep
17:46:18 sean-k-mooney i guess the import-form-copy is modifying that to add the rbd store
17:48:27 dansmith right, the glance tasks modify that property to tell us what is happening
17:52:06 sean-k-mooney dansmith: i wonder if this could be related to who owns the image
17:52:20 sean-k-mooney devstack uploads it as admin correct
17:52:33 sean-k-mooney but tempest is running with its own tenats
17:52:42 dansmith well, that's the obvious thing, but the task should be using an admin context for this kind of metadata updating
17:52:47 dansmith and they say it should
17:52:49 sean-k-mooney so perhaps they do not have permission to modify that porperty
17:53:22 sean-k-mooney that would be the logical thing to do yes
17:53:29 sean-k-mooney but manybet its not
17:53:58 sean-k-mooney https://github.com/openstack/glance/commit/1754c9e2b085ba0fc37a4369488c92a40268997a add the copy image support so im just skiming it quickly to see what it does
18:01:23 sean-k-mooney home ok i dont see how the propery gets updated in that but i also dont know how glance works internally so its not suprising.
18:01:32 sean-k-mooney oh time for a call...
18:19:53 dansmith melwitt: a while back I asked about getting admin credentials for glance and you pointed me to something I ignored because I decided I didn't need admin
18:20:03 dansmith melwitt: do you remember that and if so can you point me again?
18:26:35 melwitt heh, sec
18:29:19 melwitt dansmith: it might have been this commit https://github.com/openstack/nova/commit/aab4b7a0e2504c04e08389145bcb1414dea63631
18:29:58 melwitt just as an example of a place where we needed to use an admin cred to make a particular API call
18:34:08 dansmith melwitt: okay that's just a flag to the neutron client right?
18:34:47 dansmith I thought there was something more general
18:37:46 melwitt dansmith: yeah, I think when I linked you I was just saying, it is normal/expected for us to have to selectively use admin to call other APIs and that was a recent example of us doing it
18:38:33 dansmith oh, okay, that's common in a lot of places, yeah.. what I need is a way to get admin creds to talk to glance
18:38:54 dansmith I don't really know how we do that for neutron.. I think long ago we had credentials in our config, but that's gone now right?
18:41:28 melwitt I don't know off the top of my head. I thought we did have creds but I don't know about them being gone. I'm looking through the code now to see if it's obvious
18:42:34 dansmith I thought there was some service user thing we use now, but yeah I don't really know
18:53:03 melwitt based on this code block, there are supposed to be creds used from nova.conf https://github.com/openstack/nova/blob/f1ebc15dfc8ffb7f23b2cb9879f0ca9376931a90/nova/network/neutron.py#L191
19:01:42 melwitt and here's a config file from a nova-next run showing what look to be service user creds for neutron and placement https://zuul.opendev.org/t/openstack/build/785733b6379b40a5982f710a62302c21/log/controller/logs/etc/nova/nova_cell1_conf.txt#40
19:07:26 dansmith melwitt: sorry in three conversations here
19:07:38 dansmith melwitt: yeah, okay, I thought we had moved past that at some point, but it looks like not
19:07:40 melwitt np. I'm still gathering info
19:08:16 melwitt we implemented this https://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/use-service-tokens.html which says it should have docs for setting up the service user stuff in conf but I don't find any docs so far
19:08:23 dansmith glance is kinda half requiring admin to do the image copy-to-rbd thing.. if that's intentional, then we'll need admin creds for glance too, which really sucks
19:08:29 dansmith heh
19:08:56 dansmith maybe it's done and devstack is just still using the old method?
19:10:18 melwitt I'd think that unlikely...
19:10:31 melwitt this is what I find for a change that went into devstack to enable service tokens. https://review.opendev.org/#/c/409329/8/lib/nova
19:11:07 melwitt (I'm looking through https://review.opendev.org/#/q/topic:bp/use-service-tokens)
19:11:10 dansmith melwitt: come on, be optimistic with me! :P
19:11:16 melwitt lol :)
19:11:31 melwitt well, the service token stuff involves setting creds in conf
19:11:55 melwitt for the service user
19:11:58 dansmith hmm, okay maybe the service token still means we get creds, just not creds that are general purpose admins?
19:12:04 dansmith I thought it was better than that
19:12:08 melwitt how that's different I don't really know
19:13:14 sean-k-mooney dansmith: i dont think devstack conigure the service user stuff for us by default
19:13:33 sean-k-mooney i have not looked in a while but last time i did i did not see it
19:13:55 melwitt dansmith: so.... it "seems" like you would probably do a patch similar to this one https://review.opendev.org/410394
19:14:41 melwitt that ^ adds the nova-neutron interaction. and the groundwork was added in a prior patch for the nova-cinder interaction https://review.opendev.org/397399
19:15:09 dansmith ack yeah, okay

Earlier   Later