| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-22 | |||
| 13:16:49 | sean-k-mooney | tbarron: there is no way to do that | |
| 13:16:54 | stephenfin | gibi: Yeah, I'm on the fence about that now too | |
| 13:17:08 | stephenfin | especially given the confusion it resulted in | |
| 13:17:24 | stephenfin | I don't know what dansmith and melwitt settled on when I left | |
| 13:17:29 | tbarron | sean-k-mooney: thanks, I already said "no" but thought I better check with those who really know :D | |
| 13:17:57 | gibi | stephenfin: I will try to come up with a better factored solution in the next PS | |
| 13:18:05 | stephenfin | ack | |
| 13:18:18 | sean-k-mooney | tbarron: the quickest way to enable something like that would be to write a cyborg dirver to manage disk on the host and then extend libivt to accpet disks form cyborg | |
| 13:21:17 | tbarron | sean-k-mooney: interesting, and that makes sense | |
| 13:22:59 | sean-k-mooney | tbarron: a disk is not really an acclearator but if you think of cyborg as a generic device managment service it think it fit. espacially when you consider that imaging or erasing a disk is basically the same as programing or reseting an fpga | |
| 13:23:14 | sean-k-mooney | just with less $$$ for the hardware | |
| 13:24:08 | tbarron | sean-k-mooney: yeah, it fits with the general expose-hardware-features to VMs direction, I get it | |
| 13:25:10 | tbarron | and is somewhat in tension with present compute-instances as abstractions idea, thouhh I say "tension" and not contradiction | |
| 13:26:10 | tbarron | we have a similar tension in storage where different backends have different capabilities and it's tricky figuring which to expose as abstractions | |
| 13:34:03 | sean-k-mooney | tbarron: well the tension is relived in a sense that it would be abstracted via a device-profile and we would should based on the aviablity of the resouce like any other | |
| 13:34:29 | sean-k-mooney | tbarron: cinder does nto quite workin this usecase since you dont have an ideal of a local only cinder backend | |
| 13:35:17 | dansmith | gibi: I commented | |
| 13:36:14 | dansmith | gibi: making it non-public and specific to "decide if we should report zero" addresses my original concern I guess, but I don't understand what the problem currently is | |
| 13:38:36 | gibi | dansmith: my problem that it makes a coupling between nova.virt.libvirt.imagebackend.Image.cache and nova.virt.libvirt.imagecache.ImageCacheManager.cache_dir_is_on_same_dev_as_instances_dir as the later assumes how the former will create the directory | |
| 13:39:39 | dansmith | gibi: cache_dir is a property of the imagecache no? | |
| 13:40:09 | gibi | for me the reasoning like "the directory does not exists therefore it occupies 0 space" is easier to accept than "the directory is on the same dev as it is not created but we know that when it is created it will be a call to mkdir that creates it on the same dev" | |
| 13:40:19 | dansmith | are you just saying that the behavior of creating the cache_dir if it doesn't exist is something in the libvirt code? | |
| 13:40:52 | dansmith | gibi: until the directory exists, the same exact thing is returned right? zero? | |
| 13:41:52 | tbarron | sean-k-mooney: ack, cinder volumes are intended to have a life-cycle independent of compute instances or compute instance hosts | |
| 13:41:55 | dansmith | once the directory exists, we'll report what we see, which will almost definitely be the same dev, but if not, we'll report the value according to how the directory is at that point | |
| 13:42:11 | gibi | dansmith: the behavior of get_disk_usage() is the same in my PS2 and in PS5 | |
| 13:42:49 | gibi | but I think the implementation is better strucutred in PS2 | |
| 13:42:54 | dansmith | right, so I don't see that we're making any different assumptions | |
| 13:43:04 | tbarron | sean-k-mooney: so the cinder lvm backend is useful for testing iscsi but not so much for production deployments | |
| 13:43:21 | dansmith | gibi: well, I disagree because I think that a property should explode for a known condition | |
| 13:43:37 | dansmith | gibi: but make it not a property (and rename it) and you can have that structure | |
| 13:43:48 | gibi | dansmith: I accep that I'm ready to make that an internal helper instead of a public property | |
| 13:44:00 | dansmith | I think a property shouild /not/ explode I meant | |
| 13:44:07 | gibi | yeah, I agree ^^ | |
| 13:45:09 | gibi | just to make sure I understand your point. Is it OK for you if change the property to an private helper method? | |
| 13:45:46 | dansmith | I don't like it, but it addresses the problem I had with PS2 | |
| 13:46:40 | gibi | why don't you like it? | |
| 13:49:00 | dansmith | well, because as it is, the property has utility beyond what you're doing here. You're just changing it to "should I report zero for cache" which is a single conditional and might as well just be in the if statement of the get_disk_usage() | |
| 13:49:14 | dansmith | doesn't seem worth it being a helper to me | |
| 13:49:52 | dansmith | but all I really meant is that _I_ would keep it the way it is in PS5, but it matters to me less than you, so you should change it | |
| 13:50:15 | dansmith | what matters to me is not having that should-be-useful-but-dangerous public property | |
| 13:52:42 | gibi | dansmith: thanks | |
| 14:04:43 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184 | |
| 14:25:46 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Try to make a glance multistore job https://review.opendev.org/734184 | |
| 14:27:16 | openstackgerrit | Elod Illes proposed openstack/nova stable/train: Check cherry-pick hashes in pep8 tox target https://review.opendev.org/737279 | |
| 15:08:06 | jsuchome | hey dansmith ... regular reminder about https://review.opendev.org/#/c/574301 once you have time... | |
| 15:28:00 | dansmith | jsuchome: I know, I haven't forgotten | |
| 15:45:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Guard against missing image cache directory https://review.opendev.org/736964 | |
| 16:29:46 | stephenfin | melwitt: could you look at https://review.opendev.org/708617 too? | |
| 16:30:12 | melwitt | stephenfin: sure, will do | |
| 16:30:18 | stephenfin | thanks | |
| 17:22:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Remove nova-network remnants https://review.opendev.org/737329 | |
| 17:25:56 | openstackgerrit | Ghanshyam Mann proposed openstack/nova stable/stein: Make greande jobs n-v for EM and oldest stable https://review.opendev.org/737332 | |
| 17:27:48 | openstackgerrit | Ghanshyam Mann proposed openstack/nova stable/stein: Make greande jobs n-v for EM and oldest stable https://review.opendev.org/737332 | |
| 17:34:47 | sean-k-mooney | dansmith: you can increase the job timeout in the zull.yaml if you need to for the multistore job | |
| 17:35:05 | sean-k-mooney | dansmith: it looks like you glance api change made it this time https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/etc/glance/glance-api_conf.txt | |
| 17:35:27 | dansmith | yep, and it asked for the copy | |
| 17:35:32 | dansmith | I think something else likely broke, looking now | |
| 17:35:37 | dansmith | Jun 22 15:42:22.928857 ubuntu-bionic-rax-iad-0017311577 nova-compute[23701]: INFO nova.virt.libvirt.imagebackend [None req-ca48174a-0bf4-4341-8d45-fcf69cc9a3de tempest-DeleteServersAdminTestJSON-1752858908 tempest-DeleteServersAdminTestJSON-1752858908] Asking glance to copy image e6b1a7d0-ccd8-4be3-bef7-69c68fca4313 to our rbd store robust | |
| 17:36:14 | dansmith | Jun 22 15:52:23.076886 ubuntu-bionic-rax-iad-0017311577 nova-compute[23701]: ERROR nova.compute.manager [instance: 2cb1f8e2-a6a3-4f42-b6e2-de6823c71e25] nova.exception.ImageUnacceptable: Image e6b1a7d0-ccd8-4be3-bef7-69c68fca4313 is unacceptable: Copy to store robust timed out | |
| 17:36:53 | sean-k-mooney | it might be a slow node | |
| 17:37:12 | sean-k-mooney | you could relax some of the times outs for image/volume creation | |
| 17:37:23 | dansmith | it waited ten minutes | |
| 17:37:38 | dansmith | that should be more than long enough to copy a cirros image on any node I think | |
| 17:38:03 | sean-k-mooney | ya fair point :) | |
| 17:38:33 | sean-k-mooney | i was more thinking it was a slow host becaue it hit the 2 hour job time out | |
| 17:38:46 | sean-k-mooney | althougyh i guess enough 10 minute wait would have the same effect | |
| 17:39:28 | dansmith | I think it's just because each time we went to spawn an instance, it waited ten minutes before failing, | |
| 17:39:35 | dansmith | which linearized is enough to run the timeout | |
| 17:40:11 | dansmith | https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/screen-g-api.txt#7466 | |
| 17:40:18 | dansmith | glance was failing to update its own property I think | |
| 17:40:35 | sean-k-mooney | right this si the image convertion https://zuul.opendev.org/t/openstack/build/18e4701c1a374bf09269778479160f25/log/controller/logs/screen-g-api.txt#440 | |
| 17:41:03 | sean-k-mooney | so it looks like the inital import conversion worked | |
| 17:42:04 | dansmith | the devstack conversion you mean? | |
| 17:42:14 | dansmith | had it not, nova wouldn't have even tried to boot on it, so yeah | |
| 17:42:27 | dansmith | and the devstack patch I have wouldn't have gotten past waiting for the image to go active | |
| 17:43:05 | sean-k-mooney | dansmith: yes the intial devstack conversion seam to have worked fine so the failure after after the qcow has been converted to raw and stored in teh file backedn | |
| 17:43:26 | dansmith | yep | |
| 17:43:54 | sean-k-mooney | well if nothing else i guess glance can now use your patch to test that... | |
| 17:44:43 | sean-k-mooney | os_glance_importing_to_stores seams like a strange name for a property on the image | |
| 17:45:08 | dansmith | that's the task status property | |
| 17:45:23 | sean-k-mooney | https://github.com/openstack/glance/blob/92492cf50461e214b777c707148886a8e87f340d/releasenotes/notes/import-multi-stores-3e781f2878b3134d.yaml#L25 yep | |
| 17:46:18 | sean-k-mooney | i guess the import-form-copy is modifying that to add the rbd store | |
| 17:48:27 | dansmith | right, the glance tasks modify that property to tell us what is happening | |
| 17:52:06 | sean-k-mooney | dansmith: i wonder if this could be related to who owns the image | |
| 17:52:20 | sean-k-mooney | devstack uploads it as admin correct | |
| 17:52:33 | sean-k-mooney | but tempest is running with its own tenats | |
| 17:52:42 | dansmith | well, that's the obvious thing, but the task should be using an admin context for this kind of metadata updating | |
| 17:52:47 | dansmith | and they say it should | |
| 17:52:49 | sean-k-mooney | so perhaps they do not have permission to modify that porperty | |
| 17:53:22 | sean-k-mooney | that would be the logical thing to do yes | |
| 17:53:29 | sean-k-mooney | but manybet its not | |
| 17:53:58 | sean-k-mooney | https://github.com/openstack/glance/commit/1754c9e2b085ba0fc37a4369488c92a40268997a add the copy image support so im just skiming it quickly to see what it does | |
| 18:01:23 | sean-k-mooney | home ok i dont see how the propery gets updated in that but i also dont know how glance works internally so its not suprising. | |
| 18:01:32 | sean-k-mooney | oh time for a call... | |
| 18:19:53 | dansmith | melwitt: a while back I asked about getting admin credentials for glance and you pointed me to something I ignored because I decided I didn't need admin | |
| 18:20:03 | dansmith | melwitt: do you remember that and if so can you point me again? | |
| 18:26:35 | melwitt | heh, sec | |
| 18:29:19 | melwitt | dansmith: it might have been this commit https://github.com/openstack/nova/commit/aab4b7a0e2504c04e08389145bcb1414dea63631 | |
| 18:29:58 | melwitt | just as an example of a place where we needed to use an admin cred to make a particular API call | |