Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-03
16:28:11 bauzas jaypipes: hope that clarifies ^
16:28:54 jaypipes bauzas: you don't need custom traits for any of this.
16:29:03 jaypipes bauzas: you don't need GPU types to be traits either.
16:29:16 bauzas what I want is a pony
16:29:25 mriedem exarlos: so you're hitting https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/driver.py#L7450
16:29:26 jaypipes bauzas: we had long conversations with Jianghua on the original VGPU spec about this.
16:29:32 bauzas jaypipes: what do you suggest ?
16:29:35 mriedem exarlos: which comes after https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/driver.py#L7413
16:29:49 melwitt mriedem: woot (though concerned the unit test was covering it up?)
16:29:51 mriedem exarlos: are you sure this isn't a volume-backed instance? and by volume-backed, i mean cinder volume, not local lvm disk
16:30:06 mriedem melwitt: well, the unit test was missing a check for the bdm turducken
16:30:06 jaypipes bauzas: use the traits for GPUs that are already in os-traits for describing the *capabilities* of the GPU.
16:30:12 melwitt a-ha
16:30:17 jaypipes bauzas: "nvidia-11" isn't a capability.
16:30:22 mriedem melwitt: the DriverVolumeBlockDevice wraps BlockDeviceMapping and proxies the fields to save on the internal BDM object
16:30:28 jaypipes bauzas: support for, say, OpenCL 2.0 is a capability.
16:30:39 bauzas got it, good idea
16:30:42 jaypipes bauzas: support for 4 max display heads is a capability (weirdly, but correct)
16:30:54 jaypipes bauzas: support for a particular resolution size is a capability.
16:30:57 openstackgerrit Merged openstack/os-vif stable/queens: Fix VF-rep lookup routine to use parent PF number https://review.openstack.org/554917
16:31:27 melwitt mriedem: super turducken
16:31:36 exarlos mriedem: Ohhh. So, yes, I use cinder! Sorry.
16:31:50 exarlos I thought you mean ceph or some other storage backend.
16:34:41 exarlos mriedem: Again, i'll mention that i'm running package version 15.0.7 - so could well be some way behind on the code.
16:35:11 exarlos (I don't see locally the lines you're suggesting on github)
16:37:05 mriedem exarlos: ah ok then that is why you get past this check https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L7294
16:37:17 mriedem and you fail here https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L7331
16:38:39 mriedem exarlos: and your "instances_path" config option value must be /var/lib/nova/instances right?
16:40:28 mriedem you're hitting a ProcessExecutionError on the source node during the resize, and the instance is getting put into ERROR state here https://github.com/openstack/nova/blob/stable/ocata/nova/compute/manager.py#L6792
16:40:38 mriedem the guest is otherwise fine on the source node, and you could reset it's state to ACTIVE
16:41:04 mriedem you'll have to debug why you're getting "Stderr: u"mv: cannot stat \'/var/lib/nova/instance'"
16:41:33 mriedem this is just trying to rename the directory mv /var/lib/nova/instances/97346b0a-71a5-4a66-a966-a41565cf3c80 /var/lib/nova/instances/97346b0a-71a5-4a66-a966-a41565cf3c80_resize
16:44:12 exarlos mriedem: yeah, that folder does not exist. Hence my question as to whether all VMs should have a folder in this respective /vat/lib/nova/instances folder.
16:44:21 exarlos They only contain the console.log file.
16:44:48 exarlos And only half of the folders are present, if I should expect all "servers" to have a folder here :-|
16:45:51 exarlos instances_path is default: $state_path/instances
16:46:18 mriedem and state_path is /var/lib/nova
16:47:28 exarlos mriedem: correct.
16:48:31 mriedem yes all instances should have that directory, spawn() eventually calls this https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L3104
16:49:49 mriedem hard rebooting the instance should recreate that path
16:54:00 exarlos mriedem: ah, okay. Concerns me a little bit that this folder isn't already there :-|
16:54:10 mriedem yeah, i don't know why it wouldn't be
16:55:03 mriedem although,
16:55:12 mriedem https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L7229 might have done something naughty
16:56:03 mriedem well nvm, probably not because if utils.execute('mv', inst_base, inst_base_resize) failed then inst_base_resize shouldn't exist
16:57:12 exarlos it doesn't exist. I check my other compute nodes, and they do seem to have the folders present to match their VMs, but only show a console.log
16:57:29 exarlos okay, i'll try and reset the state and fire her up.
16:58:00 mriedem they only have a console.log because the actual image data for the guest is in the root cinder volume
16:58:52 mriedem you can also just reboot the instance in ERROR state
16:59:01 mriedem and verify that it recreates the instance dir
16:59:15 mriedem it will likely have to be a HARD reboot though
16:59:20 mriedem https://developer.openstack.org/api-ref/compute/#reboot-server-reboot-action
17:03:17 exarlos mriedem: It's starting up :-)
17:04:00 exarlos And that folder exists now in /var/lib/nova/instances/ for the server.
17:05:51 exarlos I guess that using local storage for openstack is really being left behind?
17:07:45 exarlos Is ceph the go-to storage option of choice?
17:08:19 larsks Will nova permit someone to set up PCI passthrough for a device that device is already bound to an inactive (e.g. powered off) instance?
17:08:22 exarlos (for those of us who can't afford/have space at home for a real storage solution) :-)
17:10:46 mriedem exarlos: ceph is the most widely used cinder backend, i don't know how much people also use it for local ephemeral storage, but i assume if you're doing it with persistent block devices then you might as well also use it for local ephemeral
17:11:15 mriedem larsks: we don't support hot plugging pci devices
17:11:30 mriedem cyborg is the long-term solution for that i believe
17:11:58 mriedem exarlos: i'm not sure why that local disk was gone, or if it has anything to do with your storage option of choice
17:12:00 exarlos mriedem: Thankyou for your time today. I suspect you're rather busy.
17:12:05 mriedem unless you had a hard drive failure on that host or something
17:12:17 larsks mriedem: I wasn't asking about hot plugging...or was I? I didn't think so. I just meant you have instance A configure w/ pci passthrough, you shut down A and then try to create B with the same configuration...would nova allow that?
17:12:31 mriedem larsks: i hope not
17:12:35 larsks Fair enough.
17:12:40 mriedem exarlos: np, thanks and good luck
17:20:35 moshele mriedem: hi, what are the chances to approve this pike backport https://review.openstack.org/#/c/563501/ ? the packport wan't trivial at all
17:24:47 mriedem moshele: that enables the nova side of a neutron feature added in pike correct?
17:25:15 mriedem and the change on master was originally reverted https://review.openstack.org/#/c/518833/
17:25:26 mriedem master/queens
17:27:09 moshele meridem: the first version was revert because it broken heat, but the second one was merged https://review.openstack.org/#/c/519066/
17:28:18 mriedem sure, but i'm still not sure if we should backport something to nova pike to enable a feature in neutron pike
17:28:34 mriedem afk for lunch
17:30:18 moshele meridem: this is the feature enablement in pike https://review.openstack.org/#/c/398265/. the other patch was a bug fix (https://review.openstack.org/#/c/519066/ ) which was introduce by something I change in neutron
17:31:36 moshele merdem: this is the neutron change https://review.openstack.org/#/c/499203/
17:31:53 moshele mriedem: this is the neutron change https://review.openstack.org/#/c/499203/
17:32:43 moshele mriedem: ok I will try to catch you after lunchת bon appetit!
17:44:51 exarlos Thanks again, ciao for now.
17:56:18 Tahvok Hey guys!
17:58:05 Tahvok I'm having some boot issues with kvm (using ocata): After creating an instance, I configured some volumes on it, and after shutdown, I cannot bring it up, it's getting stuck in seabios with 'no bootable device' message. If I remove all other volumes I've created, it manages to boot, but fails later as my configuration needs the volumes available
17:58:45 Tahvok So I've checked the bootindex configuration, and it looks like it's completely fine, bootindex is set only on the bootable volume.
17:58:52 Tahvok I should also mention that I'm using ceph
18:00:21 mriedem Tahvok: do you have this https://review.openstack.org/#/c/561613/ ?
18:00:31 mriedem just released in 15.1.1
18:00:42 Tahvok Nova,version=15.1.1
18:01:21 Tahvok Let me check the file just to be sure
18:03:00 Tahvok So, I've just checked this, and not I do not have it
18:06:04 melwitt if you do a 'virsh dumpxml' on the instance, you might see the boot_index set incorrectly on your bootable volume (it should be 0). and if it's not, you need the bug fix from 5.1.1
18:06:52 openstack Launchpad bug 1729584 in OpenStack Compute (nova) queens "boot from volume + configdrive with virtio-scsi broken (regression)" [High,Fix committed] - Assigned to Artom Lifshitz (notartom)
18:06:52 melwitt this was the bug https://bugs.launchpad.net/nova/+bug/1729584
18:08:29 frickler mriedem: melwitt: removed the -2, will do a closer review tomorrow
18:08:51 melwitt cool, thanks frickler
18:09:09 Tahvok melwitt: yes, it's set as 1 and not 0
18:09:31 Tahvok I'm restarting nova-compute now, to check the patch
18:10:08 Tahvok And it's working!
18:10:28 melwitt sweet
18:10:33 Tahvok I wonder why it says that I have nova 15.1.1 when I definitely do not have this patch
18:11:13 melwitt hm, yeah, it should be in there

Earlier   Later