| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 13:21:20 | bauzas | mriedem: I guess we can close https://bugs.launchpad.net/nova/+bug/1427772, right? | |
| 13:21:21 | openstack | Launchpad bug 1427772 in OpenStack Compute (nova) "Instance that uses force-host still needs to run some filters" [Low,Confirmed] | |
| 13:21:40 | bauzas | mriedem: because 1/ we no longer accept to force live-migrations | |
| 13:21:49 | bauzas | 2/ we removed the CachingScheduler etc. | |
| 13:22:30 | sean-k-mooney | bauzas: we still accept force for evaucation and maybe resize/cold migrate | |
| 13:23:06 | bauzas | well, good point | |
| 13:23:08 | sean-k-mooney | im not sure if that affect the bug or not | |
| 13:23:17 | stephenfin | bauzas: It can be closed because it's possible to request a specific host without bypassing the scheduler filters | |
| 13:24:20 | sean-k-mooney | well at least the numa related stuff is going to be adress seperatly | |
| 13:24:41 | stephenfin | bauzas: Specifically, blueprint add-host-and-hypervisor-hostname-flag-to-create-server | |
| 13:25:22 | sean-k-mooney | stephenfin: that wont prevent people form using the old way with --force | |
| 13:25:29 | sean-k-mooney | using the az | |
| 13:25:41 | stephenfin | sean-k-mooney: then the answer is "you're holding it wrong" | |
| 13:26:18 | mriedem | osc by default forces live migrations to the specified host | |
| 13:26:19 | sean-k-mooney | i guess we could close it because we have a new feature to replace it | |
| 13:26:31 | sean-k-mooney | yes | |
| 13:26:33 | sean-k-mooney | well | |
| 13:26:39 | sean-k-mooney | you have to spcify a host | |
| 13:26:42 | mriedem | given the age of that bug i'd say screw it, close it | |
| 13:26:43 | sean-k-mooney | i dont know if it forces it | |
| 13:26:44 | stephenfin | we can't remove the old way and the new way was specifically added to avoid this issue, so the answer is surely use the new feature | |
| 13:26:51 | mriedem | sean-k-mooney: yes it does | |
| 13:27:05 | mriedem | osc by default (1) requires you specify a host and (2) defaults to 2.1 | |
| 13:27:08 | mriedem | which by default forces the host | |
| 13:27:19 | sean-k-mooney | :( | |
| 13:27:31 | sean-k-mooney | ok i know 1 but not 2 | |
| 13:27:36 | mriedem | that is deprecated in osc 4.0 with something i added in train | |
| 13:27:55 | mriedem | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-migrate | |
| 13:28:01 | mriedem | --live <hostname> is deprecated | |
| 13:28:15 | mriedem | but for rhosp customers using queens that's going to be an open issue for a long time | |
| 13:28:30 | sean-k-mooney | unless we backport it | |
| 13:28:45 | bauzas | anyway, nevermind | |
| 13:28:45 | sean-k-mooney | that said i dont know if we can do that | |
| 13:28:48 | stephenfin | we'll backport it, I imagine | |
| 13:29:03 | mriedem | we == red hat | |
| 13:29:10 | sean-k-mooney | well we cant backprot the api change downstream | |
| 13:29:19 | mriedem | the api change is in queens, that's not your problem | |
| 13:29:23 | mriedem | it's the client side tooling | |
| 13:29:28 | stephenfin | well you were talking about rhosp, so yes, clearly Red Hat | |
| 13:29:33 | sean-k-mooney | oh ok | |
| 13:30:20 | sean-k-mooney | i guess we should add it to our backlog to consider. the question is can we do it without breaking people | |
| 13:30:28 | sean-k-mooney | if we cant we cant backport | |
| 13:31:07 | sean-k-mooney | that said just adding --live-migration shoudl be fine | |
| 13:31:25 | stephenfin | efried, gibi, (dansmith): Interesting issue here https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@1060 | |
| 13:32:44 | stephenfin | efried, gibi, (dansmith): As the comment suggests, that test is working by creating two hosts, one of which doesn't have the second node to which a given physnet is associated https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@855 | |
| 13:33:20 | stephenfin | and we're doing that because it's not possible to have different configuration for different compute "services" in functional tests | |
| 13:33:52 | stephenfin | however, there's a check that prevents us doing just this (configured a physnet for a NUMA node that doesn't exist) that the test simply wasn't triggering | |
| 13:34:57 | stephenfin | because it's only hit as part of the calculation of the host NUMA topology object, which happens in the 'update_available_resource' periodic task, which our test wouldn't normally have time to trigger | |
| 13:35:31 | efried | not having looked yet, just reacting to ^, we have a way to trigger that periodic | |
| 13:35:39 | efried | but it's also triggered as part of instance creation fwiw | |
| 13:36:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Error out interrupted builds https://review.opendev.org/666857 | |
| 13:36:16 | stephenfin | it seems to be triggered when we query against the placement fixture https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@82 | |
| 13:36:50 | stephenfin | and that's the issue - by adding that query, we end up in a situation when the compute service for that falls over | |
| 13:37:15 | stephenfin | aaactually, maybe it was falling over already and we just never thought to check for that | |
| 13:37:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pull up compute node queries to init_host https://review.opendev.org/682680 | |
| 13:37:48 | stephenfin | which is the real reason we're getting NoValidHost - not because the compute service can't support the request but because it never actually started | |
| 13:38:51 | sean-k-mooney | stephenfin: you could check that by looking at the comptue service logs | |
| 13:39:21 | stephenfin | that's what I'm doing as we speak | |
| 13:39:22 | stephenfin | :) | |
| 13:40:24 | stephenfin | yup, it never even started \o/ | |
| 13:40:40 | sean-k-mooney | i dont know the specific incantation to do that in the functional test but i have seen mriedem and gibi asserting log messages are emmited in the functional test form different agent in the past | |
| 13:41:01 | mriedem | self.assertIn('whatever', self.stdlog.logger.output) | |
| 13:41:15 | gibi | sean-k-mooney: you can access the logs in from the logger fixture | |
| 13:41:34 | gibi | sean-k-mooney: self.stdlog.logger.output | |
| 13:41:40 | gibi | sean-k-mooney: but it is not per service | |
| 13:41:50 | sean-k-mooney | cool ya its the self.stdlog.logger.output bit that i had seen but not used personally in the past | |
| 13:46:41 | gibi | mriedem: do you think this qualifies as a rc candidate https://bugs.launchpad.net/nova/+bug/1844993 ? | |
| 13:46:42 | openstack | Launchpad bug 1844993 in OpenStack Compute (nova) "migrate a server with qos port with compute RPC pinned to 5.1 fails and leaves the qos port in an inconsistent state" [Undecided,In progress] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 13:47:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/683961 | |
| 13:47:24 | stephenfin | gibi, sean-k-mooney: ^ (mriedem too, maybe) | |
| 13:47:37 | gibi | stephenfin: looking | |
| 13:47:48 | efried | gibi: was that a regression in train? | |
| 13:48:04 | efried | (the qos one) | |
| 13:48:11 | gibi | efried: not, it is a bug in the bandwith + migration code we merged to Train | |
| 13:48:20 | mriedem | gibi: technically no because it was a regression in stein | |
| 13:48:32 | mriedem | gibi: but it would be good to get it fixed in train regardless | |
| 13:49:23 | mriedem | gibi: oh i guess that's a side effect of the other bug right? | |
| 13:50:21 | sean-k-mooney | stephenfin: so the fact we cant confiugre different compute nodes with different config has come up afew times recently | |
| 13:50:42 | sean-k-mooney | we could remove that test but should we file a bug to add support for that in the future | |
| 13:50:51 | gibi | mriedem: your migrate bug is visible if the rpc is pinned to 5.0 | |
| 13:51:01 | gibi | mriedem: the qos migrate bug is visible even if the rpc is pinned to 5.1 | |
| 13:51:20 | sean-k-mooney | stephenfin: i think there has to be a way to use mocking to make the CONF non global | |
| 13:51:59 | sean-k-mooney | and in effect then allow us to have different confics in different agents althoug set_flags likely wont work in that context | |
| 13:52:33 | mriedem | gibi: right so my bug, introduced in stein, is a problem if computes are pinned to rocky (5.0) and yours if computes are pinned to stein (5.1) | |
| 13:52:47 | mriedem | gibi: then yeah it's probably worth tagging for rc1 | |
| 13:53:03 | gibi | mriedem: OK, then we are on the same page | |
| 13:55:01 | gibi | mriedem: I will respin the fix based on your rpc api method suggestion. I haven't seen the live migration way yet | |
| 13:56:34 | mriedem | it's just a simple abstraction so the caller doesn't need to know the router.client or version internals | |
| 13:57:21 | gibi | stephenfin: theortically you can go and mock things via self.compute['test_compute1'].manager.driver and that will be service and driver selective | |
| 14:07:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948 | |
| 14:25:24 | mriedem | oh god schedule_and_build_instances is too big, have to write 100 LOC test case just to test one new line of code | |
| 14:43:05 | kashyap | stephenfin: Hey, about? | |
| 14:43:21 | kashyap | stephenfin: When you are -- | |
| 14:43:30 | stephenfin | on a call but yeah | |
| 14:43:50 | kashyap | stephenfin: How much of this is possible _today_ in Nova? https://kashyapc.fedorapeople.org/NUMA-pinning.txt | |
| 14:43:58 | kashyap | stephenfin: Sure, respond when you can | |
| 14:44:45 | stephenfin | 1 and 2 | |
| 14:44:50 | kashyap | stephenfin: Context is a QEMU dev was asking about it. | |
| 14:44:59 | stephenfin | we don't provide a way to configure IOThreads | |
| 14:45:04 | kashyap | Yeah, I see from the code that we don't do anything for IOThreads | |