Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-06
15:14:07 efried mriedem: where/how can I run a CI that exercises ironic-in-nova?
15:14:18 mriedem efried: we have a non-voting job on all nova changes
15:14:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Do not run tempest.scenario.test_network* tests in nova-next https://review.openstack.org/641416
15:14:26 mriedem ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa
15:14:47 mriedem and i think there are a couple in the experimental queue as well for multinode and neutron stuff
15:14:59 efried aha, okay.
15:15:26 efried mriedem: background: I finally bit the bullet and started hacking up a way for ironicclient to use the ksa adapter we go through all the trouble of building up on the nova side, only to throw it away.
15:15:30 sean-k-mooney efried: the other way to do it is a DNM change to ironic that depends on the nova change
15:15:51 sean-k-mooney that will runn the ironcis jobset and pull in the relevent nova patch
15:16:03 efried sean-k-mooney: and that exercises the nova side?
15:16:26 sean-k-mooney assuming they have jobs that boot ironic instances via nova
15:16:43 sean-k-mooney it just depends on what you want to test
15:16:47 efried ironicclient: https://review.openstack.org/641169
15:16:47 efried nova: https://review.openstack.org/#/c/641170/
15:16:47 efried in case you care.
15:16:49 mriedem melwitt: looks like functional test failures at the end of your counting quotas series are real https://review.openstack.org/#/c/638324/
15:17:41 mriedem efried: the one thing i'm not sure about is if that nova change will pull in the ironicclient code dependency or if it will pull in ironicclient from upper-constraints
15:17:53 mriedem might depend on if the job is zuul v3-ified
15:18:03 efried o
15:18:40 mriedem http://logs.openstack.org/70/641170/1/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/0e5cd32/controller/logs/pip2-freeze.txt.gz indicates it doesn't
15:18:44 mriedem python-ironicclient==2.6.0
15:18:47 sean-k-mooney efried: mriedem looks like the ironic base job is a zullv3 native job but it does not list openstack/nova as a required project
15:18:48 mriedem doesn't pull in the source dep
15:18:55 openstackgerrit Surya Seetharaman proposed openstack/nova master: Block automatic transport_url update for cell0 https://review.openstack.org/605414
15:19:10 efried well crap
15:19:20 mriedem so,
15:19:41 mriedem the "legacy" way to do this is a devstack change which depends on the nova/ironicclient changes and you configure devstack to pull in the client from source
15:19:47 mriedem assuming devstack has an ironic job...
15:20:09 efried and if it doesn't, make the devstack change and then make a nova change that depends on it.
15:20:13 mriedem or, your nova change depends on a devstack change that pulls in ironicclient from source which depends on the ironicclient patch
15:20:18 mriedem yeah
15:20:19 efried yeah
15:20:20 efried whee
15:20:50 mriedem https://github.com/openstack-dev/devstack/blob/master/stackrc#L345
15:21:16 sean-k-mooney efried: i can mock up a quick test patch that ill pull in one of the ironic jobs and force it to use nova and ironcic client form source if iyou like
15:21:58 efried sean-k-mooney: If you have that at your fingertips, that would be great. It would take me a while to figure out.
15:22:48 sean-k-mooney efried: if you list a project in required-project and it auto adds it to LIBS_FROM_GIT and zuul follow the depends-on stuff to pull in the requried patches
15:23:06 sean-k-mooney efried: it should only take me a minute to create one
15:38:35 openstackgerrit Jose Castro Leon proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039
15:39:24 openstackgerrit sean mooney proposed openstack/nova master: [DNM] tempest client testing jobs https://review.openstack.org/641421
15:39:37 sean-k-mooney efried: i think ^ will do what you want
15:40:42 efried ...
15:41:36 sean-k-mooney i proably should have also dropt the tox jobs but they are quick
15:41:41 efried sean-k-mooney: Cool. How is that pulling in the ironicclient patch?
15:42:17 sean-k-mooney it is based on your nova patch an your nova patch has a depends on
15:42:37 efried right, but I thought we established that the depends-on wasn't working?
15:42:57 sean-k-mooney so because i added the ironic client to the require projects it will pull it in when zuul prepares the git repos and add it ot devstack LIBS_FROM_GIT list
15:42:57 efried or is that magic instated by virtue of this having been done in .zuul.yaml?
15:43:05 efried okay, nice.
15:43:36 sean-k-mooney we should still check when it actully runs just to be sure but it should work
15:44:53 sean-k-mooney i might quickly resping to drop the tempest full and greade job too since i has not started in zuul yet
15:45:00 sean-k-mooney well its ther ebut no vms yet
15:45:09 efried sean-k-mooney: Thanks for doing that. I have plenty of work left to do on the code side, but if I can at least get it running, then I have a way to debug.
15:46:56 openstackgerrit sean mooney proposed openstack/nova master: [DNM] tempest client testing jobs https://review.openstack.org/641421
15:48:24 sean-k-mooney there we go i missed doce but that will be much more efficent
15:48:33 efried thanks sean-k-mooney
15:48:50 sean-k-mooney and no worries hopefully that works the way we want it too
15:49:44 sean-k-mooney if it does i can move those jobs into the experimental pipline or replace the current non voting jobs we have
15:49:59 mtreinish mriedem: hmm, well getting the time series graph took longer than I thought (I must be rusty): https://blog.kortar.org/wp-content/uploads/2019/03/v6_network_total-1.png
15:50:24 mtreinish we also must have dropped a test (or skip it in some configurations) which made the aggregation difficult after Sept 2018
15:50:50 mtreinish I also box and whiskered each individual test: https://blog.kortar.org/wp-content/uploads/2019/03/v6_network_scenario_run_times.png
15:51:07 mtreinish it's definitely way too slow
15:53:15 mriedem geez yeah
15:53:34 mriedem that aligns with the 2100 seconds i saw in that timed out job
15:53:51 mriedem so i posted https://review.openstack.org/#/c/641416/ as a short-term workaround for nova-next
15:54:51 mriedem i can't really decide if we should whitelist the scenario modules we care about in the nova-next job, because then new ones could be added and we'd miss them, but at the same time there is probably more we should blacklist
15:55:29 mriedem i remember the good old days when tempest took 45 minutes
15:55:35 mriedem the whole job
15:55:44 mtreinish man, that was a long time ago
15:55:45 dansmith stephenfin: can you answer my questions about the test assertions in this? https://review.openstack.org/#/c/624592
15:55:53 stephenfin dansmith: sure
15:55:56 mriedem granted, the gate was fubar'd on a weekly basis, but still
15:56:01 mriedem it was fubar'ed FAST
15:56:04 dansmith stephenfin: I was thinking it would be nice to merge another good chunk of those this cycle, but...
15:56:43 sean-k-mooney mriedem for what its work im hoping to get the os-vif gates down to that by droping the keystone and cinder apis test.
15:57:11 mtreinish keystone tests take like zero time
15:57:18 mtreinish dropping cinder will save a lot of time
15:57:45 mtreinish sean-k-mooney: if there is no integration between os-vif and cinder I was just disable cinder all together in those jobs
15:57:51 sean-k-mooney mtreinish: ya but os-vif is a lib that never talk to any other openstack service
15:57:53 mtreinish s/was/would/g
15:57:53 mriedem that's why i dropped the non-compute api tests from the nova-next job
15:58:08 sean-k-mooney mtreinish: yes im going ot
15:59:28 sean-k-mooney i think in the os-vif jobs i might keep the compute and networking api tests + the senario test for now and see what the job times look like
16:00:05 sean-k-mooney if they are under an hour ill be happy if i can get down to 45 mins or so even more so
16:02:11 mtreinish well if you disable cinder (not just blacklist the cinder api tests) it won't run a good chunk of the scenarios
16:02:37 mtreinish and those scenarios are individually among the slowest tests in a job
16:11:31 stephenfin dansmith: Done-zo https://review.openstack.org/#/c/624592/8
16:17:03 melwitt mriedem: ack, will fix. I think it's bc that patch changed things from counting instances => instance mappings, that test is probably only creating instance records and not mapping records
16:23:03 dansmith mriedem: probably need someone to make a call on breaking potentially n-net-only tests with this: https://review.openstack.org/#/c/624592
16:24:27 mriedem i know some of the converted privsep stuff is going untested because testing the privsep stuff is hard...
16:24:36 mriedem i remember wresting with trying to mock around that stuff at one point
16:24:38 mriedem i don't remember the details
16:24:49 mriedem *wrestling even
16:25:10 mriedem i'd rather not make that call today...
16:26:02 dansmith just seemed like some feature-y patches we could make independent progress on, and they were just refreshed, but.. I got stopped at the first one
16:27:00 sean-k-mooney dansmith: even in the nova-net case some of that code is likely dead code as os-vif will be used in most cases
16:28:05 dansmith sean-k-mooney: understand, but it's just hard to get on board with orphaning code with tests.. if it's dead code, then maybe we should remove it
16:28:10 mriedem in other fun news, when post live migration fails we don't set the instance to error status or reset the task_state to None apparently http://logs.openstack.org/43/635343/4/check/tempest-slow-py3/a2497ae/job-output.txt.gz#_2019-03-06_05_29_44_910801
16:28:18 sean-k-mooney the gateway code is likely still used
16:30:12 sean-k-mooney mriedem: ya i have seen that happen a few times

Earlier   Later