| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-03 | |||
| 16:40:38 | mriedem | the guest is otherwise fine on the source node, and you could reset it's state to ACTIVE | |
| 16:41:04 | mriedem | you'll have to debug why you're getting "Stderr: u"mv: cannot stat \'/var/lib/nova/instance'" | |
| 16:41:33 | mriedem | this is just trying to rename the directory mv /var/lib/nova/instances/97346b0a-71a5-4a66-a966-a41565cf3c80 /var/lib/nova/instances/97346b0a-71a5-4a66-a966-a41565cf3c80_resize | |
| 16:44:12 | exarlos | mriedem: yeah, that folder does not exist. Hence my question as to whether all VMs should have a folder in this respective /vat/lib/nova/instances folder. | |
| 16:44:21 | exarlos | They only contain the console.log file. | |
| 16:44:48 | exarlos | And only half of the folders are present, if I should expect all "servers" to have a folder here :-| | |
| 16:45:51 | exarlos | instances_path is default: $state_path/instances | |
| 16:46:18 | mriedem | and state_path is /var/lib/nova | |
| 16:47:28 | exarlos | mriedem: correct. | |
| 16:48:31 | mriedem | yes all instances should have that directory, spawn() eventually calls this https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L3104 | |
| 16:49:49 | mriedem | hard rebooting the instance should recreate that path | |
| 16:54:00 | exarlos | mriedem: ah, okay. Concerns me a little bit that this folder isn't already there :-| | |
| 16:54:10 | mriedem | yeah, i don't know why it wouldn't be | |
| 16:55:03 | mriedem | although, | |
| 16:55:12 | mriedem | https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L7229 might have done something naughty | |
| 16:56:03 | mriedem | well nvm, probably not because if utils.execute('mv', inst_base, inst_base_resize) failed then inst_base_resize shouldn't exist | |
| 16:57:12 | exarlos | it doesn't exist. I check my other compute nodes, and they do seem to have the folders present to match their VMs, but only show a console.log | |
| 16:57:29 | exarlos | okay, i'll try and reset the state and fire her up. | |
| 16:58:00 | mriedem | they only have a console.log because the actual image data for the guest is in the root cinder volume | |
| 16:58:52 | mriedem | you can also just reboot the instance in ERROR state | |
| 16:59:01 | mriedem | and verify that it recreates the instance dir | |
| 16:59:15 | mriedem | it will likely have to be a HARD reboot though | |
| 16:59:20 | mriedem | https://developer.openstack.org/api-ref/compute/#reboot-server-reboot-action | |
| 17:03:17 | exarlos | mriedem: It's starting up :-) | |
| 17:04:00 | exarlos | And that folder exists now in /var/lib/nova/instances/ for the server. | |
| 17:05:51 | exarlos | I guess that using local storage for openstack is really being left behind? | |
| 17:07:45 | exarlos | Is ceph the go-to storage option of choice? | |
| 17:08:19 | larsks | Will nova permit someone to set up PCI passthrough for a device that device is already bound to an inactive (e.g. powered off) instance? | |
| 17:08:22 | exarlos | (for those of us who can't afford/have space at home for a real storage solution) :-) | |
| 17:10:46 | mriedem | exarlos: ceph is the most widely used cinder backend, i don't know how much people also use it for local ephemeral storage, but i assume if you're doing it with persistent block devices then you might as well also use it for local ephemeral | |
| 17:11:15 | mriedem | larsks: we don't support hot plugging pci devices | |
| 17:11:30 | mriedem | cyborg is the long-term solution for that i believe | |
| 17:11:58 | mriedem | exarlos: i'm not sure why that local disk was gone, or if it has anything to do with your storage option of choice | |
| 17:12:00 | exarlos | mriedem: Thankyou for your time today. I suspect you're rather busy. | |
| 17:12:05 | mriedem | unless you had a hard drive failure on that host or something | |
| 17:12:17 | larsks | mriedem: I wasn't asking about hot plugging...or was I? I didn't think so. I just meant you have instance A configure w/ pci passthrough, you shut down A and then try to create B with the same configuration...would nova allow that? | |
| 17:12:31 | mriedem | larsks: i hope not | |
| 17:12:35 | larsks | Fair enough. | |
| 17:12:40 | mriedem | exarlos: np, thanks and good luck | |
| 17:20:35 | moshele | mriedem: hi, what are the chances to approve this pike backport https://review.openstack.org/#/c/563501/ ? the packport wan't trivial at all | |
| 17:24:47 | mriedem | moshele: that enables the nova side of a neutron feature added in pike correct? | |
| 17:25:15 | mriedem | and the change on master was originally reverted https://review.openstack.org/#/c/518833/ | |
| 17:25:26 | mriedem | master/queens | |
| 17:27:09 | moshele | meridem: the first version was revert because it broken heat, but the second one was merged https://review.openstack.org/#/c/519066/ | |
| 17:28:18 | mriedem | sure, but i'm still not sure if we should backport something to nova pike to enable a feature in neutron pike | |
| 17:28:34 | mriedem | afk for lunch | |
| 17:30:18 | moshele | meridem: this is the feature enablement in pike https://review.openstack.org/#/c/398265/. the other patch was a bug fix (https://review.openstack.org/#/c/519066/ ) which was introduce by something I change in neutron | |
| 17:31:36 | moshele | merdem: this is the neutron change https://review.openstack.org/#/c/499203/ | |
| 17:31:53 | moshele | mriedem: this is the neutron change https://review.openstack.org/#/c/499203/ | |
| 17:32:43 | moshele | mriedem: ok I will try to catch you after lunchת bon appetit! | |
| 17:44:51 | exarlos | Thanks again, ciao for now. | |
| 17:56:18 | Tahvok | Hey guys! | |
| 17:58:05 | Tahvok | I'm having some boot issues with kvm (using ocata): After creating an instance, I configured some volumes on it, and after shutdown, I cannot bring it up, it's getting stuck in seabios with 'no bootable device' message. If I remove all other volumes I've created, it manages to boot, but fails later as my configuration needs the volumes available | |
| 17:58:45 | Tahvok | So I've checked the bootindex configuration, and it looks like it's completely fine, bootindex is set only on the bootable volume. | |
| 17:58:52 | Tahvok | I should also mention that I'm using ceph | |
| 18:00:21 | mriedem | Tahvok: do you have this https://review.openstack.org/#/c/561613/ ? | |
| 18:00:31 | mriedem | just released in 15.1.1 | |
| 18:00:42 | Tahvok | Nova,version=15.1.1 | |
| 18:01:21 | Tahvok | Let me check the file just to be sure | |
| 18:03:00 | Tahvok | So, I've just checked this, and not I do not have it | |
| 18:06:04 | melwitt | if you do a 'virsh dumpxml' on the instance, you might see the boot_index set incorrectly on your bootable volume (it should be 0). and if it's not, you need the bug fix from 5.1.1 | |
| 18:06:52 | openstack | Launchpad bug 1729584 in OpenStack Compute (nova) queens "boot from volume + configdrive with virtio-scsi broken (regression)" [High,Fix committed] - Assigned to Artom Lifshitz (notartom) | |
| 18:06:52 | melwitt | this was the bug https://bugs.launchpad.net/nova/+bug/1729584 | |
| 18:08:29 | frickler | mriedem: melwitt: removed the -2, will do a closer review tomorrow | |
| 18:08:51 | melwitt | cool, thanks frickler | |
| 18:09:09 | Tahvok | melwitt: yes, it's set as 1 and not 0 | |
| 18:09:31 | Tahvok | I'm restarting nova-compute now, to check the patch | |
| 18:10:08 | Tahvok | And it's working! | |
| 18:10:28 | melwitt | sweet | |
| 18:10:33 | Tahvok | I wonder why it says that I have nova 15.1.1 when I definitely do not have this patch | |
| 18:11:13 | melwitt | hm, yeah, it should be in there | |
| 18:11:40 | melwitt | 5.1.1 was released yesterday, very recent | |
| 18:12:28 | Tahvok | So I definitely do not have this | |
| 18:12:49 | Tahvok | As we rerun ansible a week ago | |
| 18:13:34 | melwitt | do you build packages yourself from stable/ocata? or where did you get the packages? | |
| 18:13:45 | Tahvok | I'm using openstack-ansible project | |
| 18:14:29 | melwitt | okay. I'm not familiar with how it works but I'm guessing maybe it builds packages off stable as the "next version" which would have been 5.1.1 | |
| 18:14:51 | melwitt | that is, if you were to run it today it would give you 5.1.2, just a guess | |
| 18:15:19 | Tahvok | Yes, it's uses hashes from git | |
| 18:15:34 | Tahvok | Ok, so that might be why I see 5.1.1 | |
| 18:15:39 | eandersson | Random question but why does the scheduler need to log every time it does an instance sync? | |
| 18:15:57 | mriedem | eandersson: need to? | |
| 18:16:02 | Tahvok | I will request to bump the hashes so I'll get this fix 'officially' | |
| 18:16:04 | mriedem | because if thinks you care | |
| 18:16:09 | eandersson | :D | |
| 18:16:11 | mriedem | *it | |
| 18:16:17 | eandersson | It's just with 800+ nodes it gets a bit spammy :D | |
| 18:16:19 | melwitt | Tahvok: ah, k. yeah. so when we released 5.1.1, that means it was tagged as 15.1.1 and pushed to pypi etc as that version | |
| 18:16:24 | mriedem | i had a patch for oslo.log at one point to be able to drop stuff like that in the scheduler to AUDIT log | |
| 18:16:26 | mriedem | below DEBUG | |
| 18:16:43 | mriedem | eandersson: is it debug today? | |
| 18:16:53 | eandersson | INFO | |
| 18:16:56 | melwitt | Tahvok: this was the release patch. I'm not sure if/how openstack-ansible consumes that https://review.openstack.org/564044 | |
| 18:17:02 | eandersson | but we are still on mitaka (so might have changed) | |
| 18:17:22 | mriedem | eandersson: which specific message? | |
| 18:17:40 | mriedem | Successfully synced instances from host ? | |
| 18:18:04 | mriedem | yeah those should probably be debug | |
| 18:18:14 | eandersson | Yea | |
| 18:18:27 | mriedem | the "received <operation> from unknown host" is probably ok at INFO | |
| 18:18:42 | mriedem | but all the other "OH HI I DID SOMETHING I"M SUPPOSED TO DO!!!" is debug | |