| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-19 | |||
| 08:02:18 | gibi | good morning nova | |
| 08:03:45 | gibi | stephenfin: hi! dansmith +2 all over the qos remaining patches, could you check back to those? https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports-ussuri | |
| 08:04:32 | gibi | stephenfin: the major change since you looked at is a compute service version check in the API | |
| 08:05:16 | gibi | to ensure the computes are on Ussuri version before we start moving the servers around as the feauture needs support from the compute service | |
| 08:05:37 | gibi | due to the PCI claim magic | |
| 08:55:53 | openstackgerrit | Lee Yarwood proposed openstack/nova master: gate: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | |
| 08:56:29 | nightmare_unreal | hello is there specific channel for python-openstackclient | |
| 09:00:34 | brinzhang_ | It seems there is not, you can send to ML, and add [openstackclient] [your_project] tag | |
| 09:00:46 | brinzhang_ | nightmare_unreal | |
| 09:28:29 | nightmare_unreal | thanks | |
| 09:43:11 | luyao | dansmith, artom: Thanks for your comments on https://review.opendev.org/#/c/687856, I replied on them, and thanks for alex_xu helping me clarifying. | |
| 09:43:29 | frickler | nightmare_unreal: brinzhang_: #openstack-sdks | |
| 09:44:23 | luyao | artom: FYI. I also run your live migration functional tests on top of my patch, they all passed | |
| 10:20:48 | openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 10:22:43 | openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 10:30:41 | openstackgerrit | Merged openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035 | |
| 11:01:12 | lyarwood | wow it finally landed | |
| 11:01:47 | openstackgerrit | Lee Yarwood proposed openstack/nova master: gate: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | |
| 11:03:14 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | |
| 11:03:15 | lyarwood | ^ sorry just trying to get this some additional runs before review to ensure we don't need the VIF timeout again | |
| 11:03:54 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713836 | |
| 11:04:12 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713837 | |
| 11:05:43 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713840 | |
| 11:05:43 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713839 | |
| 11:07:44 | kashyap | lyarwood: Why sorry? | |
| 11:07:52 | kashyap | lyarwood: You're doing the donkey work, people better be grateful ;-) | |
| 11:08:05 | kashyap | I mean to say it in a _good sense_, BTW! | |
| 11:10:42 | lyarwood | thanks :) | |
| 11:10:48 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713844 | |
| 11:10:48 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Stop setting glance_api_version in cinder.conf in nova-live-migration https://review.opendev.org/713843 | |
| 11:10:49 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713845 | |
| 11:15:14 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036 | |
| 11:15:15 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713848 | |
| 11:34:23 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | |
| 11:34:42 | brinzhang_ | damsmith: I have replied your question in https://review.opendev.org/#/c/693828/ | |
| 12:27:13 | openstackgerrit | Johannes Kulik proposed openstack/nova master: Don't recompute weighers' minval/maxval attributes https://review.opendev.org/713863 | |
| 12:28:18 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713848 | |
| 12:28:19 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036 | |
| 12:34:35 | openstackgerrit | Johannes Kulik proposed openstack/nova master: Don't recompute weighers' minval/maxval attributes https://review.opendev.org/713863 | |
| 12:53:44 | openstackgerrit | Merged openstack/nova master: images: Allow the output format of qemu-img info to be controlled https://review.opendev.org/706898 | |
| 13:12:03 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) queens "nova needs to disallow resource consumption changes on image rebuild" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 13:12:03 | lyarwood | sean-k-mooney: when you're around, PetrTuma had some quesitons about https://bugs.launchpad.net/nova/+bug/1763766 and stable/rocky | |
| 13:12:17 | sean-k-mooney | lyarwood: i can take a look now | |
| 13:12:37 | PetrTuma | hello, hold on :) | |
| 13:12:46 | PetrTuma | I'm wondering about state of https://bugs.launchpad.net/nova/+bug/1763766 in Rocky release, from the bug report it would seem Rocky should be affected by this (there is fix proposed to Rocky). I'm trying to reproduce the issue, but no matter what combination of flavors / images with different sets of properties I use, the resource consumption is | |
| 13:12:47 | PetrTuma | nearly filled up to its capacity by NUMA instances, because the NUMATopologyFilter returns empty set of hosts). I'm not sure what I might be doing wrong that I don't see the resource change during rebuild or did I miss some special condition? | |
| 13:12:47 | openstack | Launchpad bug 1804502 in OpenStack Compute (nova) rocky "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 13:12:47 | PetrTuma | not changed at all during rebuild (all properties on image I'm rebuilding to are ignored). This is follow up on https://bugs.launchpad.net/nova/+bug/1804502. We had to include the fix in our Rocky (disable the filter on rebuild), because we were having the issue similar to the one described in the bug report (i.e. rebuild failing on compute host | |
| 13:13:32 | sean-k-mooney | PetrTuma: during a rebuil you should _NOT_ be able to change the resouce usage | |
| 13:14:01 | sean-k-mooney | so if you should not be able to rebuild from an image with 1 numa node to 2 numa nodes | |
| 13:14:11 | sean-k-mooney | or form a pinned immage to an unpined one | |
| 13:14:53 | sean-k-mooney | so the fact that the topology you are rebuilding too is ignored is partly correct | |
| 13:15:19 | sean-k-mooney | the fix on master is to block any rebuild attpemnt that would alter the numa toplogy requested | |
| 13:15:44 | sean-k-mooney | so with this backported the expected behavior would be that the rebuild is rejected by the api | |
| 13:16:41 | sean-k-mooney | we can only fix https://bugs.launchpad.net/nova/+bug/1804502 | |
| 13:16:42 | openstack | Launchpad bug 1804502 in OpenStack Compute (nova) rocky "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 13:17:03 | openstack | Launchpad bug 1763766 in OpenStack Compute (nova) queens "nova needs to disallow resource consumption changes on image rebuild" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 13:17:03 | sean-k-mooney | by adressing https://bugs.launchpad.net/nova/+bug/1763766 and explictily blocking the case where the toplogy would change | |
| 13:17:16 | sean-k-mooney | in doing so we can disabel the numa toplogy filter safely on rebuild | |
| 13:17:40 | sean-k-mooney | it is not correct to disabel the numa toplogy filter on rebuild without blocking numa toplogy changes first | |
| 13:17:43 | PetrTuma | well, that's the thing, if I try to rebuild instance with hw_numa_node=1 to image with hw_numa_node=2 I have same result, resources of the instance are unchanged (this is the case where I don't have the API change to block this in my nova) | |
| 13:17:48 | sean-k-mooney | PetrTuma: does ^ make sense | |
| 13:18:15 | sean-k-mooney | PetrTuma: yes that is becasue rebuild uses a noop claim | |
| 13:18:28 | sean-k-mooney | so the resouces calimed on the host are not updated | |
| 13:19:05 | sean-k-mooney | but we considerd it to be a bug to allow you to requst a numa toplogy and not honour that requst | |
| 13:19:12 | sean-k-mooney | which is why we block it | |
| 13:19:47 | PetrTuma | yes, I understand that, I was trying to actually somehow recreate the issue, but so far I failed utterly | |
| 13:20:15 | sean-k-mooney | well recreating is simple you just boot a vm with 1 numa node and rebuild to 2 | |
| 13:20:42 | sean-k-mooney | on rocky without any backport that will result in the image stating the vm has 2 numa nodes but it will only have 1 | |
| 13:20:53 | sean-k-mooney | until you cold migrate and then it will change to two | |
| 13:21:24 | sean-k-mooney | if you live migrate it will schdule based on 2 claime 2 and pin to one. | |
| 13:21:49 | sean-k-mooney | so live migration wont fix the inconsetnecy and will like cause others | |
| 13:22:24 | PetrTuma | ah, so that's what I misunderstood | |
| 13:22:55 | PetrTuma | it's not the issue with rebuild itself, but any subsequent action that might be run on the instance | |
| 13:23:08 | sean-k-mooney | yes | |
| 13:23:29 | sean-k-mooney | cold migate and shelve/unshevel could fix the instnace. as would evacuate i guess | |
| 13:23:47 | sean-k-mooney | but live migate, hard reboot rescue would not help | |
| 13:24:21 | PetrTuma | ok, thanks for clarification, I'm going to run some more tests then (I have the fix backported to my nova) | |
| 13:24:49 | sean-k-mooney | sure let me know if you have any specific questions. | |
| 13:25:44 | sean-k-mooney | we technically could allow rebuild to change resouce usage but its invaive as we would have to alter rebuild to not use a noop claim and instead use a move claim. basicaly allowing a rebuild to change host | |
| 13:25:57 | sean-k-mooney | like a resize | |
| 13:26:32 | lyarwood | sean-k-mooney: https://review.opendev.org/#/c/713674/ is ready for review again btw, had to remember to escapte a few things. | |
| 13:27:03 | sean-k-mooney | that is not something that would be backportable and im not sure its something we want to do | |
| 13:27:09 | sean-k-mooney | lyarwood: cool | |
| 13:27:35 | sean-k-mooney | so ya the virsh list was not evaluated on the target host | |
| 13:27:43 | sean-k-mooney | i was assuming it would not be | |
| 13:28:05 | sean-k-mooney | look like the job passed which is a good sign | |
| 13:28:29 | lyarwood | yeah I haven't seen it fail yet since we stopped all of the devstack@* services on the subnode | |
| 13:29:04 | lyarwood | I also don't see the ceph errors in n-cpu on the primary host | |
| 13:29:33 | sean-k-mooney | presumably because the vm is actully stopped on the source host | |
| 13:29:39 | lyarwood | yeah | |
| 13:29:48 | lyarwood | just confirming | |
| 13:31:57 | sean-k-mooney | lyarwood: ya so i think that looks good | |
| 13:32:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Use domain capabilities to get supported device models https://review.opendev.org/666915 | |
| 13:32:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775 | |
| 13:33:07 | sean-k-mooney | https://review.opendev.org/#/c/713342/1 isnt doing what i wanted it too. i might take a look at it again but i think im going to rebase it on top of your change | |
| 13:33:35 | sean-k-mooney | im seeing some port not fond messages with it | |
| 13:33:58 | stephenfin | gibi, alex_xu: Addressed your comments in that disk image metadata series ^ | |
| 13:34:11 | sean-k-mooney | which makes me think its doing more then removing the binding host | |
| 13:34:47 | gibi | stephenfin: ack | |