| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-03 | |||
| 16:19:55 | efried | bauzas: Sure, that works. | |
| 16:20:15 | bauzas | efried: ok, what I need is testing then | |
| 16:20:30 | bauzas | efried: my patch series is close to provide inventories using update_provider_tree() | |
| 16:20:44 | efried | bauzas: There's no change to the placement API for any of that; the only thing we did was update the u-p-t spec/docs to make it clear that a dance is required to make sure you're not stepping on each other. | |
| 16:21:12 | efried | bauzas: Oh, are you aware of https://review.openstack.org/#/c/560444/ ? | |
| 16:21:12 | bauzas | efried: ok, will look | |
| 16:21:40 | bauzas | efried: no, I wasn't | |
| 16:21:54 | bauzas | efried: my top patch in the series returns a dict of dicts | |
| 16:22:08 | bauzas | efried: I'll use that dict of dicts to populate the tree | |
| 16:22:31 | bauzas | efried: but I was considering to implement by a change the new public interface for libvirt | |
| 16:22:34 | openstackgerrit | Merged openstack/nova master: Get anchors for sharing providers https://review.openstack.org/565279 | |
| 16:22:42 | openstackgerrit | Merged openstack/nova master: Make association_refresh configurable https://review.openstack.org/565526 | |
| 16:23:00 | efried | bauzas: Have you been following the xen driver work for vgpus? | |
| 16:23:08 | bauzas | efried: barely but yes | |
| 16:23:10 | efried | bauzas: I think they're the furthest along with using upt in their driver. | |
| 16:23:30 | bauzas | efried: well, I'm pretty close too | |
| 16:23:35 | efried | bauzas: Of course, they're based on code that hasn't merged and/or won't work until the nrp-in-alloc-cands series lands. | |
| 16:23:55 | bauzas | efried: last change in my series is https://review.openstack.org/#/c/564215/ | |
| 16:24:13 | bauzas | efried: I know, and I was about to propose your series for a runway slot | |
| 16:24:42 | bauzas | I'm a bit out of review as of those days due to my patch series, but I can switch to reviewing if I'm blocked due to that series :p | |
| 16:25:06 | bauzas | anyway, I need to drop for today | |
| 16:25:08 | efried | bauzas: Granular is ready as well. (Except both series are going to bounce on merge conflict soon) | |
| 16:25:32 | bauzas | granular is already a runway candidate | |
| 16:25:32 | jaypipes | bauzas: I'm a strong -2 for having standard traits (or really, any traits) representing GPU vendor types | |
| 16:25:33 | efried | I guess I could put both of those series into the runway queue. | |
| 16:25:45 | efried | jaypipes: eh?? | |
| 16:25:50 | bauzas | jaypipes: I'm cool with that | |
| 16:25:57 | exarlos | mriedem: pastebin.com/SPpHu9Tu is the current error state of the VM. | |
| 16:26:11 | bauzas | jaypipes: I guess your concern is the os-traits explosion | |
| 16:26:14 | melwitt | looks like the gate is timing out a lot in response to me saying it had been good in the nova meeting | |
| 16:26:22 | efried | jaypipes: Oh, you're looking for GPU *features* as opposed to types. | |
| 16:26:25 | bauzas | melwitt: yup, I noticed that in the meeting too | |
| 16:26:33 | melwitt | :( sorry everyone | |
| 16:26:54 | exarlos | mriedem: I would think it's recoverable, but i'm still learning. | |
| 16:26:59 | bauzas | jaypipes: efried: anyway, I need to drop | |
| 16:27:12 | efried | bauzas: à bientôt | |
| 16:27:17 | bauzas | the first step is to make nested RPs a thing | |
| 16:27:32 | bauzas | the second step will be to test my patches using custom traits | |
| 16:27:39 | mriedem | melwitt: excellent catch on https://review.openstack.org/#/c/541420/ - i reproduced in one of the unit tests | |
| 16:28:01 | bauzas | so I'm super not concerned by having standard traits as of now, in particular for vendor purposes | |
| 16:28:11 | bauzas | jaypipes: hope that clarifies ^ | |
| 16:28:54 | jaypipes | bauzas: you don't need custom traits for any of this. | |
| 16:29:03 | jaypipes | bauzas: you don't need GPU types to be traits either. | |
| 16:29:16 | bauzas | what I want is a pony | |
| 16:29:25 | mriedem | exarlos: so you're hitting https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/driver.py#L7450 | |
| 16:29:26 | jaypipes | bauzas: we had long conversations with Jianghua on the original VGPU spec about this. | |
| 16:29:32 | bauzas | jaypipes: what do you suggest ? | |
| 16:29:35 | mriedem | exarlos: which comes after https://github.com/openstack/nova/blob/stable/ocata/nova/virt/libvirt/driver.py#L7413 | |
| 16:29:49 | melwitt | mriedem: woot (though concerned the unit test was covering it up?) | |
| 16:29:51 | mriedem | exarlos: are you sure this isn't a volume-backed instance? and by volume-backed, i mean cinder volume, not local lvm disk | |
| 16:30:06 | jaypipes | bauzas: use the traits for GPUs that are already in os-traits for describing the *capabilities* of the GPU. | |
| 16:30:06 | mriedem | melwitt: well, the unit test was missing a check for the bdm turducken | |
| 16:30:12 | melwitt | a-ha | |
| 16:30:17 | jaypipes | bauzas: "nvidia-11" isn't a capability. | |
| 16:30:22 | mriedem | melwitt: the DriverVolumeBlockDevice wraps BlockDeviceMapping and proxies the fields to save on the internal BDM object | |
| 16:30:28 | jaypipes | bauzas: support for, say, OpenCL 2.0 is a capability. | |
| 16:30:39 | bauzas | got it, good idea | |
| 16:30:42 | jaypipes | bauzas: support for 4 max display heads is a capability (weirdly, but correct) | |
| 16:30:54 | jaypipes | bauzas: support for a particular resolution size is a capability. | |
| 16:30:57 | openstackgerrit | Merged openstack/os-vif stable/queens: Fix VF-rep lookup routine to use parent PF number https://review.openstack.org/554917 | |
| 16:31:27 | melwitt | mriedem: super turducken | |
| 16:31:36 | exarlos | mriedem: Ohhh. So, yes, I use cinder! Sorry. | |
| 16:31:50 | exarlos | I thought you mean ceph or some other storage backend. | |
| 16:34:41 | exarlos | mriedem: Again, i'll mention that i'm running package version 15.0.7 - so could well be some way behind on the code. | |
| 16:35:11 | exarlos | (I don't see locally the lines you're suggesting on github) | |
| 16:37:05 | mriedem | exarlos: ah ok then that is why you get past this check https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L7294 | |
| 16:37:17 | mriedem | and you fail here https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L7331 | |
| 16:38:39 | mriedem | exarlos: and your "instances_path" config option value must be /var/lib/nova/instances right? | |
| 16:40:28 | mriedem | you're hitting a ProcessExecutionError on the source node during the resize, and the instance is getting put into ERROR state here https://github.com/openstack/nova/blob/stable/ocata/nova/compute/manager.py#L6792 | |
| 16:40:38 | mriedem | the guest is otherwise fine on the source node, and you could reset it's state to ACTIVE | |
| 16:41:04 | mriedem | you'll have to debug why you're getting "Stderr: u"mv: cannot stat \'/var/lib/nova/instance'" | |
| 16:41:33 | mriedem | this is just trying to rename the directory mv /var/lib/nova/instances/97346b0a-71a5-4a66-a966-a41565cf3c80 /var/lib/nova/instances/97346b0a-71a5-4a66-a966-a41565cf3c80_resize | |
| 16:44:12 | exarlos | mriedem: yeah, that folder does not exist. Hence my question as to whether all VMs should have a folder in this respective /vat/lib/nova/instances folder. | |
| 16:44:21 | exarlos | They only contain the console.log file. | |
| 16:44:48 | exarlos | And only half of the folders are present, if I should expect all "servers" to have a folder here :-| | |
| 16:45:51 | exarlos | instances_path is default: $state_path/instances | |
| 16:46:18 | mriedem | and state_path is /var/lib/nova | |
| 16:47:28 | exarlos | mriedem: correct. | |
| 16:48:31 | mriedem | yes all instances should have that directory, spawn() eventually calls this https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L3104 | |
| 16:49:49 | mriedem | hard rebooting the instance should recreate that path | |
| 16:54:00 | exarlos | mriedem: ah, okay. Concerns me a little bit that this folder isn't already there :-| | |
| 16:54:10 | mriedem | yeah, i don't know why it wouldn't be | |
| 16:55:03 | mriedem | although, | |
| 16:55:12 | mriedem | https://github.com/openstack/nova/blob/15.0.7/nova/virt/libvirt/driver.py#L7229 might have done something naughty | |
| 16:56:03 | mriedem | well nvm, probably not because if utils.execute('mv', inst_base, inst_base_resize) failed then inst_base_resize shouldn't exist | |
| 16:57:12 | exarlos | it doesn't exist. I check my other compute nodes, and they do seem to have the folders present to match their VMs, but only show a console.log | |
| 16:57:29 | exarlos | okay, i'll try and reset the state and fire her up. | |
| 16:58:00 | mriedem | they only have a console.log because the actual image data for the guest is in the root cinder volume | |
| 16:58:52 | mriedem | you can also just reboot the instance in ERROR state | |
| 16:59:01 | mriedem | and verify that it recreates the instance dir | |
| 16:59:15 | mriedem | it will likely have to be a HARD reboot though | |
| 16:59:20 | mriedem | https://developer.openstack.org/api-ref/compute/#reboot-server-reboot-action | |
| 17:03:17 | exarlos | mriedem: It's starting up :-) | |
| 17:04:00 | exarlos | And that folder exists now in /var/lib/nova/instances/ for the server. | |
| 17:05:51 | exarlos | I guess that using local storage for openstack is really being left behind? | |
| 17:07:45 | exarlos | Is ceph the go-to storage option of choice? | |
| 17:08:19 | larsks | Will nova permit someone to set up PCI passthrough for a device that device is already bound to an inactive (e.g. powered off) instance? | |
| 17:08:22 | exarlos | (for those of us who can't afford/have space at home for a real storage solution) :-) | |
| 17:10:46 | mriedem | exarlos: ceph is the most widely used cinder backend, i don't know how much people also use it for local ephemeral storage, but i assume if you're doing it with persistent block devices then you might as well also use it for local ephemeral | |
| 17:11:15 | mriedem | larsks: we don't support hot plugging pci devices | |