| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-11-23 | |||
| 11:14:22 | sean-k-mooney | you have /var/lib/nova/instances/compute_nodes | |
| 11:14:27 | sean-k-mooney | *do you have | |
| 11:14:55 | gokhani | yes I have | |
| 11:15:08 | sean-k-mooney | does it have muliple entries | |
| 11:15:32 | gokhani | I will check it | |
| 11:15:45 | sean-k-mooney | mine looks like this | |
| 11:15:47 | sean-k-mooney | nova-compute)[nova@cloud instances]$ cat /var/lib/nova/instances/compute_nodes | |
| 11:15:49 | sean-k-mooney | {"cloud": 1669200576.0961165} | |
| 11:16:03 | sean-k-mooney | im not sure if that should have muliple entries for nfs | |
| 11:16:30 | sean-k-mooney | but that is created by the image cache code and i think its related ot deleteion on shared filesystmes | |
| 11:16:56 | sean-k-mooney | im just wondering if it exists and if it has multipel entires the content itself is not really that important | |
| 11:18:00 | gokhani | sean-k-mooney, https://paste.openstack.org/show/bldgeGSIq8BoLjp0u6sx/ | |
| 11:18:28 | sean-k-mooney | ack that is what i was expecting to see | |
| 11:19:15 | sean-k-mooney | that generated here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/storage_users.py#L45-L72 | |
| 11:19:46 | sean-k-mooney | so each compute on shared storage adds themselves to that list | |
| 11:21:24 | sean-k-mooney | https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L10901-L10916 | |
| 11:21:50 | sean-k-mooney | we use that file to get the instnace for all host on the shared storage | |
| 11:21:59 | sean-k-mooney | and tehn we clean the cache based on that | |
| 11:22:14 | sean-k-mooney | on a normal non shared deployment like mine that only has one host | |
| 11:22:20 | sean-k-mooney | in your case it has many | |
| 11:23:22 | sean-k-mooney | i see prod-compute3 but not compute03 in that list | |
| 11:23:40 | gokhani | sean-k-mooney, we need to find how image backing file is deleted . I have this problem on instances whose images are deleted on glance | |
| 11:24:15 | sean-k-mooney | the delete proably happen because of the perodic cleanup | |
| 11:24:44 | sean-k-mooney | but can you verify if the compute node is compute03 or prod-compute3 in the hypervior api or compute_node db table | |
| 11:25:29 | sean-k-mooney | we are using conf.host to populate that file | |
| 11:25:42 | sean-k-mooney | im wondering if you perhaps changed that at some point? | |
| 11:25:48 | sean-k-mooney | or change the hostname | |
| 11:26:02 | sean-k-mooney | to remove the prod- prefix | |
| 11:27:20 | sean-k-mooney | my guess is that currently eitehr the [DEFAULT]/host value does not match the value in the compute_nodes file or its unset and your hostname nolonger matches because it was change somehow | |
| 11:28:01 | sean-k-mooney | that file is updated before every time we cleanup the image cache | |
| 11:28:13 | gokhani | sean-k-mooney, https://paste.openstack.org/show/bVzhWTyWsA3f1Umb5YGM/ | |
| 11:28:20 | sean-k-mooney | so the compute agents think they shoudl be usingthe prod- prefix | |
| 11:28:36 | sean-k-mooney | ok | |
| 11:28:50 | sean-k-mooney | that atleast alines to whats in the file | |
| 11:28:54 | sean-k-mooney | althoguh | |
| 11:29:06 | sean-k-mooney | thats showing the hypervior hostname | |
| 11:29:14 | gokhani | sean-k-mooney, do ı need to also verify on db ? | |
| 11:29:39 | sean-k-mooney | what i actully need is the host value rather then hypervior hostname i think | |
| 11:30:26 | sean-k-mooney | that would be the host value in the compute service entry for example although its also in teh compute nodes table | |
| 11:30:37 | sean-k-mooney | im just trying to verify that they match what is in the file | |
| 11:31:12 | sean-k-mooney | on never mind | |
| 11:31:15 | sean-k-mooney | its fine | |
| 11:31:17 | sean-k-mooney | https://paste.openstack.org/show/b7FSkdeQOxcuWu296tfq/ | |
| 11:31:33 | sean-k-mooney | i had scolled that over and th start of the lines were cut off | |
| 11:31:41 | sean-k-mooney | so its prod-compute03 | |
| 11:31:48 | gokhani | yes | |
| 11:36:07 | sean-k-mooney | do you see this warning any of the logs https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/imagecache.py#L331-L334 | |
| 11:41:07 | gokhani | sean-k-mooney, I dont see any warning like that | |
| 11:43:41 | gokhani | there are periodic checks like that https://paste.openstack.org/show/bGWC1FfgR5R7nXqiCOuQ/ | |
| 11:48:06 | sean-k-mooney | ya that what we expect to see | |
| 11:48:22 | sean-k-mooney | when its workign correctly | |
| 11:49:48 | gokhani | sean-k-mooney, for rescuing these vms it seems there is only one option create glance images from /var/lib/nova/instances/xxx/disk | |
| 11:50:28 | sean-k-mooney | glance does not allow you to create and image with a specifc uuid | |
| 11:50:53 | sean-k-mooney | breifly looking at the code i done se where the bug is | |
| 11:50:59 | sean-k-mooney | https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/imagecache.py#L43 | |
| 11:51:27 | sean-k-mooney | the list_running_isntnace code seams to take into account local and remote instnace | |
| 11:57:32 | sean-k-mooney | i have a meeting now so unfortuetly i cant continue to look at this now | |
| 11:57:52 | sean-k-mooney | the root cause so far is something deleted the backing file | |
| 11:58:18 | sean-k-mooney | since its deleted in glance and its deleted form the nfs share im not sure there is a way to correct this | |
| 11:59:45 | gokhani | thanks sean-k-mooney for your help :) | |
| 14:12:33 | opendevreview | Jorge San Emeterio proposed openstack/nova-specs master: Review usage of oslo-privsep library on Nova https://review.opendev.org/c/openstack/nova-specs/+/865432 | |
| 14:14:33 | opendevreview | Jorge San Emeterio proposed openstack/nova-specs master: Review usage of oslo-privsep library on Nova https://review.opendev.org/c/openstack/nova-specs/+/865432 | |
| 15:10:43 | opendevreview | Jorge San Emeterio proposed openstack/nova-specs master: Review usage of oslo-privsep library on Nova https://review.opendev.org/c/openstack/nova-specs/+/865432 | |
| 16:02:16 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 16:02:17 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 16:58:11 | bauzas | man, the gate is super flakey those days | |
| 16:58:29 | bauzas | lots of rechecks on networing and volume detachs :( | |
| 17:11:30 | gibi | probably the switch to jammy | |
| 17:11:51 | gibi | but this time I have not time to look at what changed with the detach logic again | |
| 17:12:04 | gibi | *I don't have time | |
| 18:28:42 | sean-k-mooney | gibi: im not sure that it really has changed | |
| 18:28:57 | sean-k-mooney | gibi: i think they still have not fixed it | |
| 18:30:42 | sean-k-mooney | gibi: bauzas by the way if eitehr of ye can approve https://review.opendev.org/c/openstack/nova/+/865031 it will simplfy ralonsoh life and help with the trunk port issue | |
| 18:31:12 | sean-k-mooney | https://review.opendev.org/c/openstack/neutron/+/837780 cuurntly need a cofnig option because of our min version | |
| 18:31:26 | sean-k-mooney | but that can be removed if we increase the min version of os-vif | |
| 19:59:35 | opendevreview | Merged openstack/nova master: Reproducer for bug 1951656 https://review.opendev.org/c/openstack/nova/+/850673 | |
| #openstack-nova - 2022-11-24 | |||
| 03:04:47 | opendevreview | Jorhson Deng proposed openstack/nova master: Remove the redundant code in HostState._update_from_compute_node https://review.opendev.org/c/openstack/nova/+/864824 | |
| 05:17:23 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 05:17:24 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 05:56:49 | opendevreview | Jorhson Deng proposed openstack/nova master: Remove the redundant code in HostState._update_from_compute_node https://review.opendev.org/c/openstack/nova/+/864824 | |
| 06:31:14 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 06:32:14 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 07:02:52 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: compute: enhance compute evacuate instance to support target state https://review.opendev.org/c/openstack/nova/+/858383 | |
| 07:02:53 | opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: api: extend evacuate instance to support target state https://review.opendev.org/c/openstack/nova/+/858384 | |
| 08:16:03 | han-guangyu | hello, cloud I ask a question | |
| 08:16:48 | han-guangyu | Could I use a windows ISO image to create instance in openstack | |
| 08:19:10 | han-guangyu | Or do I have to make an image with the virtio driver first | |
| 08:47:07 | gibi | sean-k-mooney: but we fixed it (worked around it) in tempest by waiting for the guest to boot up before we attach / detach volumes. | |
| 08:47:29 | gibi | so if tempest is failing again on master then someting is changed | |
| 09:04:40 | bauzas | gibi: yeah I need to do my homework on the flakey gate | |
| 09:05:22 | bauzas | I'm impacted for my vgpu series, I'm quite in a position where I should take time digging into the ci issue for my own egoistic sake :) | |
| 09:05:48 | bauzas | and I'm done with preparing my mock presentation for tonight's grenoble meetup :p | |
| 09:17:46 | opendevreview | Amit Uniyal proposed openstack/nova stable/ussuri: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/864006 | |
| 09:17:47 | opendevreview | Amit Uniyal proposed openstack/nova stable/ussuri: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864007 | |
| 09:25:46 | gibi | bauzas: you are in a good spot then :) | |
| 09:33:01 | bauzas | gibi: on a side note, that's fun to prescribe testing patterns to an audience with does and donts when your own project isn't that much following those testing patterns :) | |
| 09:33:27 | bauzas | like, piling our mock decorators in every single UT | |
| 09:33:59 | bauzas | I just feel schizophrenic : | |
| 09:34:00 | bauzas | :) | |
| 09:34:05 | gibi | you learned things in our project you dont want to repeate | |