| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-06 | |||
| 14:53:48 | mriedem | yeah i suppose that's all it is | |
| 14:54:43 | mtreinish | mriedem: yeah I think that's what's going on | |
| 14:54:46 | mriedem | man there are tests in tempest that really just don't belong there | |
| 14:54:47 | mriedem | tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_shutoff_status [72.172009s] ... ok | |
| 14:54:54 | mtreinish | the stackviz view is good for visualizing that: http://logs.openstack.org/72/638072/14/check/nova-next/c8ecf61/logs/stackviz/#/stdin/timeline | |
| 14:54:56 | mriedem | create a server, stop it, wait for it to be stopped | |
| 14:55:05 | HD|Laptop | hey all | |
| 14:55:08 | openstackgerrit | Jose Castro Leon proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039 | |
| 14:55:13 | mriedem | we could test shutoff server filtering in functional tests | |
| 14:56:01 | mtreinish | I'd support moving that test out of tempest | |
| 14:56:19 | HD|Laptop | how do I convince nova-compute to not include the NFS storage under "local disk store" in Horizon? | |
| 14:56:19 | mriedem | cripes these dualnet scenario tests are also long as hell | |
| 14:56:26 | mriedem | 3+ minutes | |
| 14:57:55 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Sep methods to free claimed and allocated devs https://review.openstack.org/616120 | |
| 14:58:10 | mriedem | HD|Laptop: you mean a single compute is showing local storage for the entire nfs share rather? | |
| 14:58:14 | mriedem | that's not really a horizon thing | |
| 14:58:27 | mriedem | e.g. compute1 is showing like 2TB of "local" disk | |
| 14:58:58 | HD|Laptop | mriedem: under horizon/admin/hypervisors/, each hypervisor shows with 68TB local storage | |
| 14:59:24 | HD|Laptop | probably this is because its instance_path is set to a shared folder on said NFS server | |
| 14:59:28 | mriedem | yes | |
| 14:59:48 | HD|Laptop | and horizon seems to sum up all the disks... to "Lokale Festplattenbelegung | |
| 14:59:48 | mtreinish | mriedem: iirc those tests make multiple servers and networks, login to them with an ipv4 network and test connectivity between servers over different ipv6 configurations. It takes forvever just waiting for things to come up | |
| 14:59:51 | HD|Laptop | 120GB von 582,8TB benutzt | |
| 15:00:01 | mriedem | HD|Laptop: that comes from the libvirt driver in here https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/virt/libvirt/driver.py#L5747 | |
| 15:00:11 | mriedem | https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/virt/libvirt/driver.py#L5762 | |
| 15:00:26 | mriedem | since instances_path is on shared storage, it's going to show the entire storage pool for that path | |
| 15:00:31 | mriedem | i'm not aware of a way around that | |
| 15:00:40 | HD|Laptop | ah okay. damn :'D | |
| 15:01:07 | mriedem | mtreinish: time for an "extra-slow" tag soon | |
| 15:01:41 | mriedem | mtreinish: will openstack health show me how long on average the entire TestGettingAddress class takes? | |
| 15:01:57 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: auto detach/attach sriov ports on migration https://review.openstack.org/629589 | |
| 15:02:29 | mtreinish | it doesn't let you filter the averages by class iirc | |
| 15:02:46 | mtreinish | but if you give me a sec I can hack together a graph for you manually | |
| 15:05:57 | mriedem | took ~35 minutes in logs.openstack.org/72/638072/14/check/nova-next/c8ecf61/job-output.txt.gz | |
| 15:06:00 | mriedem | 8 tests | |
| 15:14:07 | efried | mriedem: where/how can I run a CI that exercises ironic-in-nova? | |
| 15:14:18 | mriedem | efried: we have a non-voting job on all nova changes | |
| 15:14:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Do not run tempest.scenario.test_network* tests in nova-next https://review.openstack.org/641416 | |
| 15:14:26 | mriedem | ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa | |
| 15:14:47 | mriedem | and i think there are a couple in the experimental queue as well for multinode and neutron stuff | |
| 15:14:59 | efried | aha, okay. | |
| 15:15:26 | efried | mriedem: background: I finally bit the bullet and started hacking up a way for ironicclient to use the ksa adapter we go through all the trouble of building up on the nova side, only to throw it away. | |
| 15:15:30 | sean-k-mooney | efried: the other way to do it is a DNM change to ironic that depends on the nova change | |
| 15:15:51 | sean-k-mooney | that will runn the ironcis jobset and pull in the relevent nova patch | |
| 15:16:03 | efried | sean-k-mooney: and that exercises the nova side? | |
| 15:16:26 | sean-k-mooney | assuming they have jobs that boot ironic instances via nova | |
| 15:16:43 | sean-k-mooney | it just depends on what you want to test | |
| 15:16:47 | efried | ironicclient: https://review.openstack.org/641169 | |
| 15:16:47 | efried | nova: https://review.openstack.org/#/c/641170/ | |
| 15:16:47 | efried | in case you care. | |
| 15:16:49 | mriedem | melwitt: looks like functional test failures at the end of your counting quotas series are real https://review.openstack.org/#/c/638324/ | |
| 15:17:41 | mriedem | efried: the one thing i'm not sure about is if that nova change will pull in the ironicclient code dependency or if it will pull in ironicclient from upper-constraints | |
| 15:17:53 | mriedem | might depend on if the job is zuul v3-ified | |
| 15:18:03 | efried | o | |
| 15:18:40 | mriedem | http://logs.openstack.org/70/641170/1/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/0e5cd32/controller/logs/pip2-freeze.txt.gz indicates it doesn't | |
| 15:18:44 | mriedem | python-ironicclient==2.6.0 | |
| 15:18:47 | sean-k-mooney | efried: mriedem looks like the ironic base job is a zullv3 native job but it does not list openstack/nova as a required project | |
| 15:18:48 | mriedem | doesn't pull in the source dep | |
| 15:18:55 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Block automatic transport_url update for cell0 https://review.openstack.org/605414 | |
| 15:19:10 | efried | well crap | |
| 15:19:20 | mriedem | so, | |
| 15:19:41 | mriedem | the "legacy" way to do this is a devstack change which depends on the nova/ironicclient changes and you configure devstack to pull in the client from source | |
| 15:19:47 | mriedem | assuming devstack has an ironic job... | |
| 15:20:09 | efried | and if it doesn't, make the devstack change and then make a nova change that depends on it. | |
| 15:20:13 | mriedem | or, your nova change depends on a devstack change that pulls in ironicclient from source which depends on the ironicclient patch | |
| 15:20:18 | mriedem | yeah | |
| 15:20:19 | efried | yeah | |
| 15:20:20 | efried | whee | |
| 15:20:50 | mriedem | https://github.com/openstack-dev/devstack/blob/master/stackrc#L345 | |
| 15:21:16 | sean-k-mooney | efried: i can mock up a quick test patch that ill pull in one of the ironic jobs and force it to use nova and ironcic client form source if iyou like | |
| 15:21:58 | efried | sean-k-mooney: If you have that at your fingertips, that would be great. It would take me a while to figure out. | |
| 15:22:48 | sean-k-mooney | efried: if you list a project in required-project and it auto adds it to LIBS_FROM_GIT and zuul follow the depends-on stuff to pull in the requried patches | |
| 15:23:06 | sean-k-mooney | efried: it should only take me a minute to create one | |
| 15:38:35 | openstackgerrit | Jose Castro Leon proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039 | |
| 15:39:24 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] tempest client testing jobs https://review.openstack.org/641421 | |
| 15:39:37 | sean-k-mooney | efried: i think ^ will do what you want | |
| 15:40:42 | efried | ... | |
| 15:41:36 | sean-k-mooney | i proably should have also dropt the tox jobs but they are quick | |
| 15:41:41 | efried | sean-k-mooney: Cool. How is that pulling in the ironicclient patch? | |
| 15:42:17 | sean-k-mooney | it is based on your nova patch an your nova patch has a depends on | |
| 15:42:37 | efried | right, but I thought we established that the depends-on wasn't working? | |
| 15:42:57 | sean-k-mooney | so because i added the ironic client to the require projects it will pull it in when zuul prepares the git repos and add it ot devstack LIBS_FROM_GIT list | |
| 15:42:57 | efried | or is that magic instated by virtue of this having been done in .zuul.yaml? | |
| 15:43:05 | efried | okay, nice. | |
| 15:43:36 | sean-k-mooney | we should still check when it actully runs just to be sure but it should work | |
| 15:44:53 | sean-k-mooney | i might quickly resping to drop the tempest full and greade job too since i has not started in zuul yet | |
| 15:45:00 | sean-k-mooney | well its ther ebut no vms yet | |
| 15:45:09 | efried | sean-k-mooney: Thanks for doing that. I have plenty of work left to do on the code side, but if I can at least get it running, then I have a way to debug. | |
| 15:46:56 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] tempest client testing jobs https://review.openstack.org/641421 | |
| 15:48:24 | sean-k-mooney | there we go i missed doce but that will be much more efficent | |
| 15:48:33 | efried | thanks sean-k-mooney | |
| 15:48:50 | sean-k-mooney | and no worries hopefully that works the way we want it too | |
| 15:49:44 | sean-k-mooney | if it does i can move those jobs into the experimental pipline or replace the current non voting jobs we have | |
| 15:49:59 | mtreinish | mriedem: hmm, well getting the time series graph took longer than I thought (I must be rusty): https://blog.kortar.org/wp-content/uploads/2019/03/v6_network_total-1.png | |
| 15:50:24 | mtreinish | we also must have dropped a test (or skip it in some configurations) which made the aggregation difficult after Sept 2018 | |
| 15:50:50 | mtreinish | I also box and whiskered each individual test: https://blog.kortar.org/wp-content/uploads/2019/03/v6_network_scenario_run_times.png | |
| 15:51:07 | mtreinish | it's definitely way too slow | |
| 15:53:15 | mriedem | geez yeah | |
| 15:53:34 | mriedem | that aligns with the 2100 seconds i saw in that timed out job | |
| 15:53:51 | mriedem | so i posted https://review.openstack.org/#/c/641416/ as a short-term workaround for nova-next | |