Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-23
13:27:05 mriedem osc by default (1) requires you specify a host and (2) defaults to 2.1
13:27:08 mriedem which by default forces the host
13:27:19 sean-k-mooney :(
13:27:31 sean-k-mooney ok i know 1 but not 2
13:27:36 mriedem that is deprecated in osc 4.0 with something i added in train
13:27:55 mriedem https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-migrate
13:28:01 mriedem --live <hostname> is deprecated
13:28:15 mriedem but for rhosp customers using queens that's going to be an open issue for a long time
13:28:30 sean-k-mooney unless we backport it
13:28:45 bauzas anyway, nevermind
13:28:45 sean-k-mooney that said i dont know if we can do that
13:28:48 stephenfin we'll backport it, I imagine
13:29:03 mriedem we == red hat
13:29:10 sean-k-mooney well we cant backprot the api change downstream
13:29:19 mriedem the api change is in queens, that's not your problem
13:29:23 mriedem it's the client side tooling
13:29:28 stephenfin well you were talking about rhosp, so yes, clearly Red Hat
13:29:33 sean-k-mooney oh ok
13:30:20 sean-k-mooney i guess we should add it to our backlog to consider. the question is can we do it without breaking people
13:30:28 sean-k-mooney if we cant we cant backport
13:31:07 sean-k-mooney that said just adding --live-migration shoudl be fine
13:31:25 stephenfin efried, gibi, (dansmith): Interesting issue here https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@1060
13:32:44 stephenfin efried, gibi, (dansmith): As the comment suggests, that test is working by creating two hosts, one of which doesn't have the second node to which a given physnet is associated https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@855
13:33:20 stephenfin and we're doing that because it's not possible to have different configuration for different compute "services" in functional tests
13:33:52 stephenfin however, there's a check that prevents us doing just this (configured a physnet for a NUMA node that doesn't exist) that the test simply wasn't triggering
13:34:57 stephenfin because it's only hit as part of the calculation of the host NUMA topology object, which happens in the 'update_available_resource' periodic task, which our test wouldn't normally have time to trigger
13:35:31 efried not having looked yet, just reacting to ^, we have a way to trigger that periodic
13:35:39 efried but it's also triggered as part of instance creation fwiw
13:36:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: Error out interrupted builds https://review.opendev.org/666857
13:36:16 stephenfin it seems to be triggered when we query against the placement fixture https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@82
13:36:50 stephenfin and that's the issue - by adding that query, we end up in a situation when the compute service for that falls over
13:37:15 stephenfin aaactually, maybe it was falling over already and we just never thought to check for that
13:37:20 openstackgerrit Balazs Gibizer proposed openstack/nova master: Pull up compute node queries to init_host https://review.opendev.org/682680
13:37:48 stephenfin which is the real reason we're getting NoValidHost - not because the compute service can't support the request but because it never actually started
13:38:51 sean-k-mooney stephenfin: you could check that by looking at the comptue service logs
13:39:21 stephenfin that's what I'm doing as we speak
13:39:22 stephenfin :)
13:40:24 stephenfin yup, it never even started \o/
13:40:40 sean-k-mooney i dont know the specific incantation to do that in the functional test but i have seen mriedem and gibi asserting log messages are emmited in the functional test form different agent in the past
13:41:01 mriedem self.assertIn('whatever', self.stdlog.logger.output)
13:41:15 gibi sean-k-mooney: you can access the logs in from the logger fixture
13:41:34 gibi sean-k-mooney: self.stdlog.logger.output
13:41:40 gibi sean-k-mooney: but it is not per service
13:41:50 sean-k-mooney cool ya its the self.stdlog.logger.output bit that i had seen but not used personally in the past
13:46:41 gibi mriedem: do you think this qualifies as a rc candidate https://bugs.launchpad.net/nova/+bug/1844993 ?
13:46:42 openstack Launchpad bug 1844993 in OpenStack Compute (nova) "migrate a server with qos port with compute RPC pinned to 5.1 fails and leaves the qos port in an inconsistent state" [Undecided,In progress] - Assigned to Balazs Gibizer (balazs-gibizer)
13:47:13 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/683961
13:47:24 stephenfin gibi, sean-k-mooney: ^ (mriedem too, maybe)
13:47:37 gibi stephenfin: looking
13:47:48 efried gibi: was that a regression in train?
13:48:04 efried (the qos one)
13:48:11 gibi efried: not, it is a bug in the bandwith + migration code we merged to Train
13:48:20 mriedem gibi: technically no because it was a regression in stein
13:48:32 mriedem gibi: but it would be good to get it fixed in train regardless
13:49:23 mriedem gibi: oh i guess that's a side effect of the other bug right?
13:50:21 sean-k-mooney stephenfin: so the fact we cant confiugre different compute nodes with different config has come up afew times recently
13:50:42 sean-k-mooney we could remove that test but should we file a bug to add support for that in the future
13:50:51 gibi mriedem: your migrate bug is visible if the rpc is pinned to 5.0
13:51:01 gibi mriedem: the qos migrate bug is visible even if the rpc is pinned to 5.1
13:51:20 sean-k-mooney stephenfin: i think there has to be a way to use mocking to make the CONF non global
13:51:59 sean-k-mooney and in effect then allow us to have different confics in different agents althoug set_flags likely wont work in that context
13:52:33 mriedem gibi: right so my bug, introduced in stein, is a problem if computes are pinned to rocky (5.0) and yours if computes are pinned to stein (5.1)
13:52:47 mriedem gibi: then yeah it's probably worth tagging for rc1
13:53:03 gibi mriedem: OK, then we are on the same page
13:55:01 gibi mriedem: I will respin the fix based on your rpc api method suggestion. I haven't seen the live migration way yet
13:56:34 mriedem it's just a simple abstraction so the caller doesn't need to know the router.client or version internals
13:57:21 gibi stephenfin: theortically you can go and mock things via self.compute['test_compute1'].manager.driver and that will be service and driver selective
14:07:27 openstackgerrit Balazs Gibizer proposed openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948
14:25:24 mriedem oh god schedule_and_build_instances is too big, have to write 100 LOC test case just to test one new line of code
14:43:05 kashyap stephenfin: Hey, about?
14:43:21 kashyap stephenfin: When you are --
14:43:30 stephenfin on a call but yeah
14:43:50 kashyap stephenfin: How much of this is possible _today_ in Nova? https://kashyapc.fedorapeople.org/NUMA-pinning.txt
14:43:58 kashyap stephenfin: Sure, respond when you can
14:44:45 stephenfin 1 and 2
14:44:50 kashyap stephenfin: Context is a QEMU dev was asking about it.
14:44:59 stephenfin we don't provide a way to configure IOThreads
14:45:04 kashyap Yeah, I see from the code that we don't do anything for IOThreads
14:45:13 efried mriedem: what would it take to split tempest-slow-py3 into two parts?
14:45:16 stephenfin and I don't know what a PXB device is so let's assume not for that too
14:46:51 kashyap stephenfin: For that, `grep` for 'pci-expander-bus' here: https://libvirt.org/formatdomain.html
14:46:56 mriedem efried: why?
14:47:21 efried mriedem: That would make a run take ~1.5h rather than ~2.5h
14:47:29 efried that last hour is always just waiting for tempest-slow-py3 to finish.
14:47:40 openstackgerrit Merged openstack/python-novaclient master: Update master for stable/train https://review.opendev.org/683627
14:47:42 kashyap stephenfin: So the suggested IOThreads 'formula' for management tools is, _assuming_ we know the required info here:
14:47:50 mriedem efried: what do you mean the last hour is waiting for the job to finish?
14:47:54 mriedem for tempest to finish?
14:47:56 mriedem or to collect logs?
14:48:06 kashyap stephenfin: num_iothreads = min(num_devices, num_vcpus, num_host_cpus)
14:48:18 kashyap Thanks for the answer so far
14:49:24 efried mriedem: I mean that for a given patch, all the other jobs take <=1.5h to complete, but tempest-slow-py3 takes ~2.5h. Trying to reduce the wait time for a single patch to make it through CI.
14:50:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Sanity check instance mapping during scheduling https://review.opendev.org/683730
14:50:58 mriedem efried: well, there are a few options. one is to go the way of tempest-integrated-compute and make a compute-specific job that only runs compute tests we care about (sort of like nova-next) but only runs slow tests (which tempest-integrated-compute does not),
14:51:12 mriedem the other major difference is tempest-slow* is a multinode job where nova-next and tempest-integrated-compute are single node
14:51:26 mriedem the multinode + slow combo is important to run some tests that we don't otherwise run
14:51:45 mriedem one idea is just drop tempest-slow-py3 from nova's job list and make nova-next multi-node
14:51:56 mriedem nova-next runs compute api and scenario tests, including slow
14:52:04 sean-k-mooney one factor for multinode jobs is it only starts stacking the second node after the first completes
14:52:38 efried I'm sure I'm thinking of it way too simplistically, but what I was thinking was just, like, if tempest-slow-py3 currently runs 10 tests, make a tempest-slow-py3-1 that runs the first 5 and a tempest-slow-py3-2 that runs the second 5.

Earlier   Later