| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-10 | |||
| 09:52:04 | stephenfin | lyarwood: the py36 tox environment is failing for me locally (F34). I proposed this to fix it https://github.com/powervm/pypowervm/pull/17 | |
| 09:53:18 | stephenfin | We'll need that to merge and a new release of pypowervm be cut before we can get things running again. I suspect we're not seeing issues in the gate yet because an older dependency resolver that doesn't care about these things is being used there | |
| 09:53:26 | stephenfin | Just FYI since you're also using Fedora :) | |
| 10:07:46 | gibi | elodilles, lyarwood: hi! re: kernel panic in stable/victoria with volume detach: https://bugs.launchpad.net/nova/+bug/1950310 yesterday I backported the libvirt event based detach series to victoria but top of that series the kernel panic still happens (links in the bug for the recent failures) | |
| 10:08:02 | gibi | I'm out of ideas now | |
| 10:24:44 | sean-k-mooney[m] | gibi i could maybe try to revive my attempts to add support for alpine to dib so that we could use it instead of cirros | |
| 10:25:33 | sean-k-mooney[m] | i started that orignally due to kernel bugs in the cirros kernel | |
| 10:26:27 | gibi | sean-k-mooney[m]: I'm not sure if it is really cirros. i) it happens with both cirros 0.5.1 and 0.5.2 but it does not happen on master just on stable/victoria | |
| 10:26:42 | gibi | s/i)// | |
| 10:26:52 | sean-k-mooney[m] | i see | |
| 10:27:52 | sean-k-mooney[m] | if its still happening with you event based detach patch the only other thing i can think of is changing the qemu version we are using | |
| 10:29:11 | sean-k-mooney[m] | i used to maintain a devstack plugin to compile qemu form git but i have not updated it to support the mason build systme wich they now use. we could try changing the qemu we use either via a ppa or compiling it | |
| 10:29:16 | gibi | hm interesting, I don't know how different the qemu version between master and stable/victoria but I can check that | |
| 10:29:40 | sean-k-mooney[m] | master would be focal is stable victoria on bionic? | |
| 10:33:14 | frickler | gibi: sean-k-mooney[m]: if you want newer qemu, could you try running on bullseye instead ubuntu? | |
| 10:34:53 | sean-k-mooney[m] | frickler: bullseye and centos would be options. its not really about newer it could be older. if the panics are caused by a kvm or qemu bug then just using a diffent disto with different verions might fix the problem | |
| 10:36:06 | lyarwood | gibi: yeah I didn't think it would help tbh, you're still firing device_del down to QEMU and the ACPI requests to the guestOS, are we pinning tempest on stable/victoria? There was an effort recently to force tempest to wait until the guestOS was up before attaching and detaching volumes, I wonder if that logic is there? | |
| 10:36:43 | lyarwood | gibi: I've had my head downstream on something since yesterday but I'll try to help out some more for the rest of the week | |
| 10:37:27 | sean-k-mooney[m] | you mean adding the validation so that we ssh into the vm before atching/detaching | |
| 10:37:33 | sean-k-mooney[m] | or something else | |
| 10:37:55 | lyarwood | sean-k-mooney: yeah that's it | |
| 10:38:09 | lyarwood | stephenfin: ack not seen that personally but will keep it in mind | |
| 10:39:02 | sean-k-mooney[m] | lyarwood has anyone submited patches for that | |
| 10:39:20 | lyarwood | I had thought it already landed in master tbh | |
| 10:41:01 | lyarwood | okay no it hasn't | |
| 10:41:25 | lyarwood | I was sure I saw a change land for this or maybe my dreams are really becoming boring now I'm half way through my 30's | |
| 10:42:45 | sean-k-mooney[m] | we spoke about it in out inernal meeting last week but unless you wrote it i dont think it happened. | |
| 10:43:19 | sean-k-mooney[m] | if i remember correctly we just need to set validation=true on the create server line in the test | |
| 11:30:33 | gibi | lyarwood: do you have a link for the patch that adds that validation? I can add a depends-on somewhere in nova to see if that helps | |
| 12:10:10 | lyarwood | gibi: no sorry I couldn't find it, I was sure I saw something in the scenario manager for attempted to do it, I'll look again after lunch and throw something up on gerrit if I can't find it | |
| 12:11:38 | gibi | ack thanks | |
| 13:33:40 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | |
| 13:35:05 | opendevreview | Balazs Gibizer proposed openstack/placement stable/xena: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/817255 | |
| 13:35:44 | gibi | gmann: ^^ added the missing cherry-pick lines in these ^^ | |
| 14:25:39 | gmann | gibi: thanks | |
| 15:05:11 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Fix interference in db unit test https://review.opendev.org/c/openstack/nova/+/817240 | |
| 15:25:57 | opendevreview | Stephen Finucane proposed openstack/nova master: Use unittest.mock instead of third party mock https://review.opendev.org/c/openstack/nova/+/714676 | |
| 15:43:34 | lyarwood | Just when you thought the nova codebase was safe from stephenfin ;) | |
| 15:44:09 | lyarwood | Can we give you the Friday nick of SirChangeALot? | |
| 15:44:20 | stephenfin | someone complained about us still using it recently. blame that person | |
| 15:44:28 | stephenfin | whose name I can't remember | |
| 15:44:36 | lyarwood | ^_^ | |
| 15:44:43 | lyarwood | I'll queue that up later in the week | |
| 15:44:57 | dasm | "it's them, not us" :) | |
| 15:49:24 | sean-k-mooney | lyarwood: for what its worth its really pretty tirvial | |
| 15:49:45 | lyarwood | yeah I know I was only jesting | |
| 15:49:54 | sean-k-mooney | most of it was find replace and one or two other changes which are ease to fined by the change size beside the file | |
| 15:50:20 | sean-k-mooney | it made me sad you cant sort the list of files in the gerrit review by changed lines | |
| 15:50:32 | sean-k-mooney | it would be a nice feature for that type of change | |
| 15:53:01 | EugenMayer | holds that data | |
| 15:53:01 | EugenMayer | Ah nova backup process was queued, but it was queue since i had not enough space. I removed the queued images in the ui (glance images), but now instances show up a task `image backup` and i cannot see how to remove it. Is that a nove tasks i can list (cannot find anything with the nova cli) or is that rather a glance task? A little confused who | |
| 15:54:43 | EugenMayer | `glance task-list` does not show up those tassks | |
| 16:00:40 | EugenMayer | is `nova reset-state --active` the way to solve this | |
| 16:41:05 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/776250 | |
| 17:37:13 | sean-k-mooney | gibi: good to see the neutron sepc is apporved ill try to rereview https://review.opendev.org/c/openstack/nova-specs/+/787458 this week but i belive its largely the same as v9 and i was ok with the overall design a that point. | |
| 17:37:53 | sean-k-mooney | by the way when was/is the spec review day | |
| 17:38:25 | sean-k-mooney | ah the 16th | |
| 17:38:40 | sean-k-mooney | i tought i migth have missed it ill be looking at specs tomorrow in anycase | |
| 17:40:03 | sean-k-mooney | EugenMayer: reste-state shoudl be used with caution generally as a last resort before do ing a direct db modification | |
| 17:40:28 | sean-k-mooney | if its stuck in the image backup ste a hard reboot might be a better first attempt | |
| 17:40:53 | sean-k-mooney | reststate may leave the instnace in an unclean state | |
| 17:56:50 | opendevreview | Merged openstack/nova master: db: Remove models that were moved to the API database https://review.opendev.org/c/openstack/nova/+/812149 | |
| 17:57:22 | EugenMayer | sean-k-mooney that is what happened, reset state was clearing the ui, but it stuck nevertheless. Tried soft reboot, was not able to, so i rebooted the compute | |
| 17:58:00 | EugenMayer | i cannot say that openstack shines in this areas to be honest. Not used to have such undefined, oncontrolled states created by UI interactions alone | |
| 17:58:31 | EugenMayer | in general, tasks cannot be stopped, cannot be seen and cannot be controlled, while they can stuck. This is the one area i'am kind of frustrated about | |
| 17:58:56 | sean-k-mooney | EugenMayer: that is because they are not independent tasks for the most poart | |
| 17:59:30 | sean-k-mooney | there are a limit set of operation that are modeled as task like live migration that can be aborted | |
| 17:59:45 | sean-k-mooney | but most api action are not premetable once issued | |
| 18:02:01 | sean-k-mooney | EugenMayer: creating a backup for example is not really a task its an imperitive action you have taken that shoudl either complete or put the vm in an error state | |
| 18:02:18 | sean-k-mooney | that error state can be then recovered form vai a hard-reboot or other operation | |
| 18:02:50 | sean-k-mooney | where we can recover form a failed operation nova shoudl and mark the instnace event in error | |
| 18:02:52 | opendevreview | Merged openstack/nova master: objects: Remove 'bandwidth' fields from notifications https://review.opendev.org/c/openstack/nova/+/812148 | |
| 18:03:01 | opendevreview | Merged openstack/nova master: db: Remove models for removed services, features https://review.opendev.org/c/openstack/nova/+/812150 | |
| 18:03:11 | EugenMayer | actually controlling it. And if the task creating that image never reports back and most probably moves the server task/state further, it is stuck forever | |
| 18:03:11 | EugenMayer | it's what happens if 'microservice and self contained' is the main principal as an arch without controller the echo that comes with it. In an image backup, you invoke via nova, which then invokes a glance task, set some second state or whatever somewhere. So we have a glance task and state, a second task and state for the server and nothing that is | |
| 18:04:31 | opendevreview | Dan Smith proposed openstack/nova master: Log instance event wait times https://review.opendev.org/c/openstack/nova/+/817030 | |
| 18:04:38 | opendevreview | Merged openstack/nova master: db: Remove nova-network models https://review.opendev.org/c/openstack/nova/+/812151 | |
| 18:04:42 | EugenMayer | if you decouple that much, the hard thing becomes cascading and controlling events and tasks which involve several subsystem. And for this part, it seems that is where openstack has nothing in place. Nova tasks in nove - great. if it involves other parts, things get stuck very fast. Be it neutron, the storage or similar. | |
| 18:05:27 | EugenMayer | Anyway, sorry. A little frustrated. I get over it :) | |
| 18:05:36 | sean-k-mooney | we do have some recovery logic but asa an operator you really should not be managing any of that internal state | |
| 18:06:22 | sean-k-mooney | https://docs.openstack.org/api-ref/compute/#servers-actions-servers-os-instance-actions is where you can list the action performed on an instance by the way | |
| 18:06:28 | EugenMayer | (i would love to not manage it, but it becomes my duty if queued stuck because it waits to claim storage space, which does not happen and then it rolls forever) | |
| 18:07:33 | sean-k-mooney | it should actully fail if there is not enouch space. is it failing on the image upload | |
| 18:07:41 | sean-k-mooney | or the creation of the inital snapshot | |
| 18:08:56 | EugenMayer | sean-k-mooney interesting, i have see that API but assumed this was rather 'list the actions one can do'. I guess 'nova instance-action-list' is not that, but 'openstack server event list esxi` | |
| 18:09:39 | EugenMayer | well i can assure you, it did not fail but has been queued. The task was started using `nova backup ...` | |
| 18:10:29 | sean-k-mooney | qued in glance | |
| 18:10:50 | EugenMayer | Yes, but 'instance backup' in nova/compute | |
| 18:11:31 | EugenMayer | i deleted the image in glance (not stopping the task, if that is possible at all), which then stuck the compute instance in 'instance backup' forever | |
| 18:12:08 | EugenMayer | A different thing, trying to host an ESXi as a VM on openstack KVM. The sole purpose of that VM is using ESXi native tools to convert qcow2 disks to _proper_ vmdks | |
| 18:12:27 | sean-k-mooney | instance backup... so its no in image uploading or image pendign upload | |
| 18:13:03 | sean-k-mooney | EugenMayer: right we would not expect deleting the image in glance to stop the task | |
| 18:13:12 | EugenMayer | I have run this under proxmox with ethernet e1000 but under openstack it starts with virtio, which seems not to be supported by ESXi thus the VM has no interface when booted. Can i switch the interface type? | |
| 18:13:12 | sean-k-mooney | that could infact break our cleanup code | |
| 18:13:24 | EugenMayer | sean-k-mooney that i understood - after that :) | |
| 18:13:28 | sean-k-mooney | you can specify e1000 | |
| 18:13:39 | EugenMayer | via xml only? | |
| 18:13:55 | sean-k-mooney | you set hw_vif_model=e1000 in the glance image metadata | |
| 18:14:15 | sean-k-mooney | https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L111 | |
| 18:14:31 | EugenMayer | that is pure meta-data,right? | |