| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-22 | |||
| 11:38:01 | openstackgerrit | Jason SUN proposed openstack/nova master: For instances end up in cell0, set its availability_zone to None. https://review.openstack.org/618737 | |
| 11:40:45 | gary_perkins | Hi folks! General question about spinning up instances and storage: I have compute nodes with very little spare storage. Is it possible to confine all storage operations to go to Cinder? Even ephemeral storage and spinning up image-backed instances where normally local storage is allocated? Is this something flavors could handle or should handle? TIA | |
| 11:41:16 | Miouge | Is there a way to expose the compute node an instance is running on in metadata? I would have to do with DynamicJSON vendordata? | |
| 11:42:47 | gary_perkins | mriedem: Thanks for your help yesterday. The issue is still not resolved, but I'll have to return to it another time. | |
| 11:46:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239 | |
| 11:46:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240 | |
| 11:46:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 11:46:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 11:46:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend RequestGroup object for mapping https://review.openstack.org/619527 | |
| 11:46:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528 | |
| 11:46:37 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529 | |
| 12:00:43 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 12:00:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend RequestGroup object for mapping https://review.openstack.org/619527 | |
| 12:00:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239 | |
| 12:00:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528 | |
| 12:00:50 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240 | |
| 12:00:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529 | |
| 12:00:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 12:00:57 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 12:37:05 | awalende | Does anyone know how I can troubleshoot oslo_timeouts exceptions in pre-live migrations? Only 1/5 migration attempts work, all others run in somekind of oslo messaging timeout: | |
| 12:37:22 | awalende | Pre live migration failed at blies: MessagingTimeout: Timed out waiting for a reply to message ID 359a5c51e98f4474ac | |
| 12:46:33 | mhen | awalende, we're experiencing the exact same thing in one of our infrastructures - we're still investigating but had no luck identifying the cause so far | |
| 12:51:36 | awalende | We tried to cram up some migration timeout values in nova.conf but I believe it has nothing to do with the migration itself. | |
| 12:52:52 | awalende | Don't know what exactly is happening in the pre-migration. Probably networking and maybe cinder stuff... | |
| 13:01:52 | sean-k-mooney | awalende: pre-migration has a number of check and we do wire up some of the networking on the destination | |
| 13:04:06 | sean-k-mooney | awalende: the timeout could be as a result of messages getting lost or connection issue btween the compute nodes and amqp | |
| 13:05:04 | sean-k-mooney | awalende: have you seen any log messages related to services reconnecting to rabbitmq that could indicate network issues | |
| 13:06:56 | awalende | sean-k-mooney, I have enabled debug logging on the compute nodes, but the only amqp related stuff is the timeOut-Exception itself. We are running HA rabbit on 3 Controllers, I'll check there for any further infos | |
| 13:17:40 | mhen | awalende, are you using persistent queues? | |
| 13:18:20 | awalende | we don't | |
| 13:18:49 | awalende | because we had often problems with split-brain issues on rabbit | |
| 13:20:36 | mhen | good to know, then we can rule that out as the culprit for now | |
| 15:07:23 | gary_perkins | Hi folks! General question about spinning up instances and storage: I have compute nodes with very little spare storage. Is it possible to confine all storage operations to go to Cinder? Even ephemeral storage and spinning up image-backed instances where normally local storage is allocated? Is this something flavors could handle or should handle? TIA | |
| 15:11:25 | mnaser | simple and clean backport if anyone is around -- https://review.openstack.org/#/c/619349/ | |
| 15:11:51 | artom | gary_perkins, I think you want boot from volume instances | |
| 15:22:26 | gary_perkins | artom: yes, but I want to transparently enforce the use of cinder volumes for all storage so that the compute nodes don't run out of their limited capacity | |
| 15:24:46 | gary_perkins | and by, transparently, I mean, I don't want users to have to create a bootable cinder volume prior to spinning up an instance | |
| 15:41:59 | gibi | gary_perkins: have you thought about mounting a cinder volume to each compute and use that space as nova's instance directory? | |
| 15:42:58 | gary_perkins | gibi: hmmm... that's an idea! | |
| 15:45:54 | gary_perkins | gibi: thanks! I wasn't sure one could do that. I'll give that a try! | |
| 16:25:17 | artom | gibi, can you do that? I would have thought that Cinder volumes (control plane) wouldn't be accessible from the data plane (tenants) | |
| 16:25:40 | artom | gary_perkins, there was a spec for flavor-enforced boot from volume | |
| 16:25:43 | artom | Lemme find it | |
| 16:25:57 | kashyap | Hi folks, wonder if anyone have the latest, minimal (only Nova, Keystone, Glance, Neutron) DevStack local.conf. | |
| 16:26:06 | kashyap | Currently I use: http://paste.openstack.org/show/735950/ | |
| 16:26:25 | artom | kashyap, wouldn't just putting enabled_services=blah be sufficient? | |
| 16:26:31 | artom | The defaults have worked for me | |
| 16:26:46 | kashyap | artom: That's what I do: ENABLED_SERVICES=g-api,g-reg,key,n-api,n-cpu,placement-api,n-sch,n-cond,mysql,rabbit,dstat,quantum,q-svc,q-agt,q-dhcp,q-l3,q-meta | |
| 16:27:05 | artom | kashyap, then I don't think you need anything else? | |
| 16:27:09 | kashyap | I don't want all the needless crap to be installed on the system. It's difficult to cut crap; but far easier to start minimal | |
| 16:27:57 | artom | gary_perkins, https://review.openstack.org/#/c/511965/ | |
| 16:27:59 | kashyap | artom: Yeah, that's what I normally go with. Writing up a bunch of scripts that setup live migration w/ non-shared storage from scratch in nested envs | |
| 16:28:19 | gary_perkins | thanks artom :) | |
| 16:28:21 | artom | gary_perkins, abandoned because of lack of operator/user interest. Perhaps you could be the one drumming up support ;) | |
| 16:30:24 | gibi | artom: as far as I understand cinder volumes passed to a VM are also attached to the compute host and then the resulting device is passed to the VM via libvirt | |
| 16:30:30 | gibi | artom: but I can be worng | |
| 16:30:31 | gibi | wrong | |
| 16:32:39 | artom | gibi, oh, right, yeah, I... I'm actually not sure what I understood from your idea | |
| 16:32:58 | artom | For some reason I thought the VMs would be mounting the volumes inside the guest OS | |
| 16:36:04 | jackding | jaypipes: Hi Jay, I have responded to your question on https://review.openstack.org/#/c/618542/ | |
| 16:38:06 | gibi | artom: my idea is to use cinder volumes as local disk providers for the compute host directlty | |
| 16:38:48 | artom | gibi, right, so you'd mount the nova instances directly over iscsi or NFS or whatever from the volume's connection_info | |
| 16:39:24 | artom | I guess? I'm not sure how supported that is. It seems weird to do that instead of deploying Ceph or something explicitly for this purpose | |
| 16:40:34 | gibi | artom: yeah deploy ceph would be the ideal solution, or using the cinder backend directly | |
| 16:40:59 | artom | gibi, actually yeah, why not just use Cinder's backend directly. | |
| 16:41:46 | artom | Hrmm, would there be a way to set a compute node's disk capacity to 0? | |
| 16:42:02 | artom | It wouldn't make boot from volume transparent, but it would at least prevent local storage from being used. | |
| 16:42:19 | gary_perkins | artom: +1 | |
| 16:43:11 | gary_perkins | yeah, I could just manually setup iscsid/tgt or user cinder. I just didn't know one could use cinder outside of OpenStack | |
| 16:44:06 | artom | gary_perkins, not Cinder itself directly, I think. But Cinder's backend is what gibi was suggesting. | |
| 16:44:38 | gary_perkins | artom: ahhh, in my case that's just LVM | |
| 16:44:40 | artom | gary_perkins, also, what version is this? Because you could also set disk_allocation_ratio to something really tiny to effectively reserve the vast majority of the computes's disks | |
| 16:44:58 | gary_perkins | I'm running Queens | |
| 16:45:48 | gary_perkins | because Rocky doesn't yet work on the Arm machines I'm using | |
| 16:45:48 | artom | That disk_allocation_hack would work with Queens, I think | |
| 16:45:55 | gibi | artom: instead of allocation ratio you can use reserved_host_disk_mb to reserve disk | |
| 16:46:03 | gary_perkins | artom: thanks, I'll look into it | |
| 16:46:11 | artom | gibi, hah, thanks, yeah, that's smarter :) | |
| 16:55:13 | openstackgerrit | Matthew Booth proposed openstack/nova master: Workaround a race initialising version control in db_version() https://review.openstack.org/619622 | |
| #openstack-nova - 2018-11-23 | |||
| 00:56:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: Re-propose numa-aware-live-migration spec https://review.openstack.org/599587 | |
| 01:23:21 | openstackgerrit | Yikun Jiang proposed openstack/nova master: Remove live_migration_progress_timeout config https://review.openstack.org/619142 | |
| 02:00:39 | alex_xu | stephenfin: do you know whether we support resize for numa topology stuff? I didn't see where we update the numa topo in req_spec with new flavor when resize | |
| 06:32:37 | openstackgerrit | Yikun Jiang proposed openstack/nova master: Remove live_migration_progress_timeout config https://review.openstack.org/619142 | |
| 06:49:54 | openstackgerrit | Merged openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/614757 | |
| 07:47:17 | openstackgerrit | Yongli He proposed openstack/nova-specs master: add 'show-server-group' spec https://review.openstack.org/612255 | |
| 07:51:24 | openstackgerrit | Yongli He proposed openstack/nova-specs master: add spec "show-server-numa-topology" https://review.openstack.org/612256 | |
| 09:01:45 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Bump compute service to indicate attach/detach root volume is supported https://review.openstack.org/614750 | |
| 09:37:28 | lyarwood | mdbooth: morning, can I bug you about fake_imagebackend this morning? | |
| 09:37:37 | mdbooth | lyarwood: You may | |
| 09:39:06 | lyarwood | mdbooth: cool, so, https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/fake_imagebackend.py#L124-L183 and the way it's currently returning a func over an actual class | |
| 09:39:32 | lyarwood | mdbooth: do you recall why you wired it up this up instead of returning some fake Image class? | |
| 09:39:43 | mdbooth | Have I got a patch up which changes that? | |
| 09:39:59 | mdbooth | The issue, iirc, is that you'd need to have a closed class | |
| 09:40:07 | mdbooth | i.e. the function it returns is a closure | |
| 09:40:19 | lyarwood | ahhhhh | |
| 09:40:25 | mdbooth | If you returned a class, the class would also have to be a closure over the same data | |
| 09:40:41 | mdbooth | However, I vaguely recall I may have a patch which touches this | |
| 09:41:44 | lyarwood | right, so I have that WIP change on top of another series from you where I'm attempting to use the static is_shared_block_storage method from the image classes that I obviously can't when it's a func | |
| 09:41:47 | mdbooth | Hmm, if I do I can't see it right now | |
| 09:42:53 | mdbooth | I'm *convinced* I've written a patch for this already | |