| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-02 | |||
| 12:13:16 | lyarwood | it's direct if you're using that codepath | |
| 12:13:20 | sean-k-mooney | yes | |
| 12:13:32 | sean-k-mooney | but its not when not using the hyperv driver | |
| 12:13:42 | sean-k-mooney | and we said we were not going to do an os-brick bump for that | |
| 12:13:43 | lyarwood | we'd never bump os-brick with that logic | |
| 12:14:06 | lyarwood | there have been plenty of examples in the past where we have done this to accomidate the libvirt driver | |
| 12:14:18 | lyarwood | I don't get how that's optional | |
| 12:14:26 | sean-k-mooney | true however it makes all lib deps viral | |
| 12:14:31 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Add support for microversion v2.88 https://review.opendev.org/c/openstack/python-novaclient/+/770573 | |
| 12:15:17 | sean-k-mooney | if a lib bumps there min then any project that bumps there min to pick up that new lib version is forced to transitivly pick up any min verion change in that lib or its deps | |
| 12:15:49 | lyarwood | right and that's in LC at present | |
| 12:16:04 | sean-k-mooney | this did not use to happen with the old resolver | |
| 12:16:12 | lyarwood | because it was broken | |
| 12:16:19 | sean-k-mooney | no i know how it worked | |
| 12:16:23 | sean-k-mooney | it was not broken | |
| 12:16:31 | sean-k-mooney | it just worked differently then we expected | |
| 12:16:42 | sean-k-mooney | and that had some useful cases | |
| 12:16:48 | sean-k-mooney | it had some negitives too | |
| 12:17:08 | sean-k-mooney | but we have regressed our ablity to state teh min verion of nova requirements with the new resolver | |
| 12:17:41 | sean-k-mooney | now we can only state if you deploy every possible backend connently on the same host this is the set of all min deps | |
| 12:17:49 | sean-k-mooney | that is a very different statement | |
| 12:18:25 | sean-k-mooney | previously we only needed to do a bump it we use a python api that did not exist or had a different signiture | |
| 12:19:12 | sean-k-mooney | if nova did not use a feature in a trasitive dep directly we did not need to bump it in lc | |
| 12:19:24 | sean-k-mooney | for optional deps that was the correct behaivor | |
| 12:21:58 | lyarwood | I'm not disagreeing, but given the current behaviour of pip and the fact that we can't limit deps to each backend I don't see what choice we had here? | |
| 12:22:13 | sean-k-mooney | we could not do the bump | |
| 12:22:18 | lyarwood | land code that we know would fail with our current reqs? | |
| 12:22:26 | sean-k-mooney | there is no code path in any of our test coverage that will require this | |
| 12:22:43 | lyarwood | there is in third party for hyperv | |
| 12:22:48 | sean-k-mooney | lyarwood: it wont break our unit or funct test | |
| 12:22:56 | lyarwood | and? | |
| 12:23:00 | sean-k-mooney | lyarwood: right and they can install the dep | |
| 12:23:03 | lyarwood | we're still landing broken code | |
| 12:23:09 | lyarwood | it just feels wrong to me | |
| 12:23:25 | lyarwood | and in any case bumping os-brick like this pulls in a load of other fixes and improvements | |
| 12:23:28 | sean-k-mooney | so does importing deps form an optional lib | |
| 12:23:36 | lyarwood | I'd argue that we needed to do that this cycle anyway tbh | |
| 12:24:04 | lyarwood | there are no new deps there btw, just version bumps | |
| 12:24:06 | sean-k-mooney | we can do that for other reasons but i think this is a bad pattern to apply in the general case | |
| 12:24:13 | sean-k-mooney | oh i know | |
| 12:24:15 | lyarwood | the same would also apply | |
| 12:24:23 | sean-k-mooney | but we dont want every lib to be viral | |
| 12:24:41 | sean-k-mooney | you also have a smaller set of changes then the hyperv patch orginally had | |
| 12:25:07 | lyarwood | yeah I'm not sure where some of the changes in the original hyperv change came from but pip was happy with these | |
| 12:25:47 | lyarwood | aaaaaand something has already failed | |
| 12:25:53 | sean-k-mooney | if we want to proceed with this for this cycle i wont try to block it but i do think we need to have a ptg discusssion about this and i dont think we should do this in the future | |
| 12:26:23 | sean-k-mooney | the new resolver has changed the meanin of LC and expanded its scope | |
| 12:26:47 | sean-k-mooney | if we want to continue to use that i think we need to discuss what LC now means | |
| 12:27:32 | lyarwood | sean-k-mooney: yeah I've added a line in the ptg pad but feel free to rephrase the question | |
| 12:29:06 | sean-k-mooney | lyarwood: run tox with -r to recreate the env | |
| 12:29:16 | sean-k-mooney | or before that check the pip version | |
| 12:29:37 | sean-k-mooney | it wont automatically upgrade the pip version | |
| 12:29:52 | sean-k-mooney | so if the enve was created with the old resolver then it will still be using it | |
| 12:30:39 | sean-k-mooney | oh | |
| 12:30:43 | lyarwood | that's a different job | |
| 12:30:50 | sean-k-mooney | lyarwood: you forgot to update requirements.txt | |
| 12:30:52 | lyarwood | I'm just about to run it locally now | |
| 12:31:00 | sean-k-mooney | you only bumped the min in lc | |
| 12:31:09 | lyarwood | for the in-direct deps? | |
| 12:31:13 | lyarwood | I thought that wasn't required? | |
| 12:31:17 | sean-k-mooney | it is | |
| 12:31:30 | sean-k-mooney | we also use those directly | |
| 12:31:30 | lyarwood | huh pip was fine without them | |
| 12:31:44 | sean-k-mooney | pip is but this is the requiremetns check job | |
| 12:32:10 | sean-k-mooney | we dont allow the min in requirements.txt to differ form lc by policy in openstack | |
| 12:32:47 | sean-k-mooney | and that job enforces that while also check the markers match what is in GR/UC | |
| 12:33:28 | lyarwood | *sigh* | |
| 12:33:29 | sean-k-mooney | lyarwood: the script that does this is in the requiremetns repo if i rememebr correctly its not in nova so you cant test this with tox in nova | |
| 12:33:42 | lyarwood | yeah just building the venv now | |
| 12:34:10 | sean-k-mooney | it wont see this error since this is not part of nova | |
| 12:34:54 | lyarwood | the requirements venv | |
| 12:34:59 | lyarwood | not nova | |
| 12:35:02 | sean-k-mooney | ah | |
| 12:35:28 | sean-k-mooney | anyway its a simple fix | |
| 12:35:37 | sean-k-mooney | just update nova's requiremetns.txt | |
| 12:35:44 | sean-k-mooney | with the same min version you set in lc | |
| 12:36:46 | sean-k-mooney | you do not need to change anything in the requiremetns repo in case that is not obvious form the error | |
| 12:37:46 | lyarwood | it's obvious, I just wanted to run the same test locally | |
| 12:37:53 | lyarwood | from the requirements repo | |
| 12:43:05 | ahsen | Hi, I'm getting an error while creating an instance. It says "Build of instance xxx aborted: Volume xxx did not finish being created even after we waited 188 seconds or 61 attempts. And its status is downloading." I did not get this error before and there is'nt any error on Cinder's logs. Do you have any idea why am I getting this error? How can I | |
| 12:43:06 | ahsen | increase waiting time or attemps? We are using Ussuri. Thank you. | |
| 12:46:36 | lyarwood | ahsen: the retries and interval are controlled by CONF.block_device_allocate_retries and CONF.block_device_allocate_retries_interval on the Nova side | |
| 12:46:50 | lyarwood | ahsen: but you should trace the request through to the Cinder side to understand why it's taking so long | |
| 12:52:09 | sean-k-mooney | ahsen: is it a large image or are you using HDDs on the cinder side or low bandwith nics | |
| 12:53:04 | sean-k-mooney | ahsen: i had to set block_device_allocate_retries_interval=10 on my home cluster | |
| 12:53:55 | sean-k-mooney | the time it took for qemu image to copy the image data for larger images was taking just over 60 seconds and it was timing out | |
| 12:54:17 | ahsen | lyarwood Thank you, I will try to increase those values. And actually Cinder creates volumes but I don't know how long does it take | |
| 12:55:10 | sean-k-mooney | ahsen: if you look at teh cidner driver log you should see the qemu-img command doing the data transfer | |
| 12:55:38 | sean-k-mooney | ahsen: for me it only became a proable for images over about 5-8Gs | |
| 12:56:53 | ahsen | sean-k-mooney Image is not large and we are using SSD also bandwith is not low | |
| 12:57:44 | lyarwood | ahsen: kk, you should see a request-id logged by Nova from Cinder that you can use to grep through your logs | |
| 12:57:54 | sean-k-mooney | ok increasing the interval to 10 might help but you should look at the cidner logs and try and determin why its taking so long in that case | |
| 12:58:12 | sean-k-mooney | ahsen: what cinder backend are you using by the way | |
| 12:58:38 | ahsen | sean-k-mooney I will look at them | |
| 12:58:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: requirements.txt: Bump os-brick to 4.2.0 https://review.opendev.org/c/openstack/nova/+/778177 | |
| 12:58:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: hyper-v rbd volume support https://review.opendev.org/c/openstack/nova/+/763550 | |
| 12:58:56 | lyarwood | okay lets try this again | |
| 13:00:28 | ahsen | lyarwood and sean-k-mooney I will try what you said. Thank you both | |
| 13:11:52 | openstackgerrit | Merged openstack/nova master: libvirt: add AsyncDeviceEventsHandler https://review.opendev.org/c/openstack/nova/+/772381 | |
| 13:26:44 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/673790/14/nova/virt/libvirt/host.py@1244 - going to grab some lunch but let me know if that concern isn't clear still. | |